Monorepo Guide
The Morio source code is hosted in the certeu/morio repository on GitHub, and a read-only mirror is available on code.europa.eu. Morio uses a so-called , meaning that it combines the source code for different aspects of the Morio project in a single repository.
This guide will help you familiarise yourself with the repository layout, where you can find what, as well as how to use the repository locally.
Prerequisites
To use our monorepo, you need git and NodeJS, as well as a GitHub account. To run Morio locally in development mode, you also need Docker.
If you have those, you can skip ahead to getting started. If not, read on.
Installing git
Git is the de facto standard version control system for software development. It is what we use to keep the Morio source code under version control.
If you are new to git, or feel uncomfortable about using it, there is no need to worry, we’ve all been there.
To help you get over git’s learning curve, we have created a git training course that explains how it works, and helps you getting things done. Chapter 7 deals specifically with Installing git.
It comes highly recommended: Git Training.
- If you are on linux, git is probably already installed, and if not you should
be able to install it from your distribution’s package manager. One of
apt install git
,yum install git
, ordnf install git
should do the trick. - If you are on Mac, git comes with the Xcode developer tools, so if you have Xcode installed, you are good to go. If not, install it.
- If you are on Windows, then download the git installer for windows and follow the steps.
Once installed, type git
in a terminal window.
If you get a bunch of output, we are good to go.
The instructions on this page should work for both Linux and Mac systems.
Windows users using the Linux Subsystem for Windows
(LSW) will probably
be OK, but we do not test or support this.
Install NodeJS
Morio uses NodeJS, a JavaScript runtime.
You may already have NodeJS on your system. Run node
in a command line
terminal to test that.
If you do not have NodeJS, we recommend using nvm — short for Node Version Manager — to install it as that will make it easy to install NodeJS and use different versions.
To setup nvm, follow the install instructions in the nvm README.
Once installed, type nvm
in a terminal window.
If you get a bunch of output, nvm was installed successfully.
Install NodeJS lts/iron
Morio currently uses NodeJS 20 which is known as lts/iron
.
The LTS releases (lts) provide long term support and are recommended for production use.
To install the lts/iron
release of NodeJS with nvm, you run this command:
nvm install lts/iron
When it’s all done, running node -v
in your terminal should confirm that
version 20 of NodeJS is installed and ready to go.
Installing Docker
If you want to run Morio locally, you also need Docker.
- If you are on linux, you should be able to install docker
from your distribution’s package manager. Debian-based distros should only
need to run
apt install docker.io
.
RedHat pushes Podman hard, so their derivatives typically also come with Podman. Which is no good because Morio does its own orchestration, so it needs the Docker daemon. You cannot run Morio on Podman. Make sure to install Docker. - If you are on Mac, you will need to install Docker Desktop
- If you are on Windows, then you cannot run Morio locally.
Create a GitHub account
If you do not have a GitHub account, you can create one by going to github.com/signup and following the steps.
GitHub accounts are free of charge — you do not have to pay anything.
Getting started
To get started, we will clone the repository on your local system.
You can either clone the repository directly or first fork it, then clone the forked repository:
Clone the repository directly if:
- You do not want to make any changes, or
- You have write access to the
certeu/morio
repository
Clone a fork of the repository if:
- You want to make changes, and
- You do not have write access to the
certeu/morio
repository
Fork the repository
To fork the repository, you need to be logged in to GitHub, then navigate to github.com/certeu/morio/fork.
This will create a copy of the certeu/morio
repository under your own account.
Given that it is your own copy, you will be allowed to make changes in it.
Clone the repository
If you did fork the certeu/morio
repository, replace certeu/morio
by
username/morio
, where username
is your GitHub username.
Open a terminal window, and navigate to a place where you are happy to keep your git content.
If you are not sure where to put it, let us create a git
folder in your home
directory and enter it:
mkdir ~/git
cd ~/git
Now we can run the command to clone and enter the repository:
git clone git@github.com:certeu/morio.git
cd morio
Run kickstart
Once the repository is cloned, you should run the kickstart
run script:
npm run kickstart
It will install dependencies and configure your local repository for use.
When it’s done, your local Morio repository is ready.
Repository layout
The repository has a number of top-level files and folders. The most relevant are:
- Files:
CHANGELOG.md
: A list of changes in Morio over time and versionsCODE_OF_CONDUCT.md
: Morio’s Code of ConductLICENSE
: Holds the Morio license (EUPL)package.json
: The NodeJS configuration filepackage-lock.json
: The NodeJS dependency lock fileREADME.md
: Holds general info about the repository, this is what’s shown on github.com/certeu/morioSECURITY.md
: Holds instructions on how to report security issues
- Hidden Files:
.editorconfig
: Holds Morio’s EditorConfig configuration.eslintignore
: Holds a list of files/patterns ESLint should ignore.gitignore
: Holds a list of files/patterns git should ignore.prettierignore
: Holds a list of files/patterns Prettier should ignore.prettierrc.json
: Holds the Prettier configuration
- Folders:
builders
: Holds source code for the various Morio builder servicesclients
: Holds source code for the various Morio clientsconfig
: Holds configurationcore
: Holds source code for the core servicedocs
: Holds (this) documentationmedia
: Holds images and other mediamoriod
: Holds source code for the moriod packagenode_modules
: Holds installed NodeJS dependenciesschema
: Holds configuration for the data schema used in Morioscripts
: Holds various scripts to automate working with the repositoryshared
: Holds shared Morio codeui
: Holds the source code of the UI service
- Hidden Folders
.git
: Holds git’s internal data.github
Holds GitHub-specific configuration, such as the CI pipelines to run.husky
: Holds the git pre-commit hook configuration
Branches and merge requests
Morio’s default branch is develop
whereas the main
branch is considered the
production branch.
Both branches are protected. To propose changes, you must open a pull request to
the develop
branch (the default branch). The pull request can be made from
your own fork, or from a feature branch in the certeu/morio
, assuming you
have the rights to create branches there.
Pull requests will start a number of CI pipelines. Please make sure that all pipelines pass before requesting review.
Before a merge request can be merged, it must get a positive review from someone on the Morio team. You can request reviews from github users joostdecock, serge-tellene or lilianbaz.
Running the Morio development environment
Depending on what you are looking to work on or contribute to, and your local environment, you may want to:
- Run the documentation site
- Run Morio
- Use a FQDN to set up Morio
- Run Morio on MacOS
- Run Morio without network access
Running the documentation site
To run the development version, enter the docs
folder and run npm run start
:
cd docs
npm run start
This will open the documentation site at http://localhost:3000
When you make changes, the page will be updated automatically, no reload needed.
Running Morio
To run Morio itself, you need to fist build the development version of the various container images. In the monorepo root, run this command:
cd docs
npm run build
When that works, you can start Morio with:
npm run dev
Open your browser at https://localhost and — after telling your browser to ignore the invalid certificate — will see the Morio setup screen.
Create an FQDN for Morio
If you want to set up Morio on your local machine, you need an that resolves to a routable IP address that points at your machine.
The easiest way to accomplish that is to use a service like nip.io which provides DNS records for any IP address.
For example, if your local machine’s IP address is currently 192.168.1.4
you
can use 192-168-1-4.nip.io
as FQDN.
Now, connect to https://192-168-1-4.nip.io/ and setup a standalone
Morio node using 192-168-1-4.nip.io
as FQDN.
If you have a DNS zone you control, you can also create a record yourself.
/etc/hosts
?You could update /etc/host
to add the IP/FQDN mapping, but that requires some
extra steps. Refer to Run Morio without network
access for details.
Run Morio on MacOS
Morio uses the journald
log driver for Docker, which is not available
on MacOS. If you run npm run dev
you will get an error from the
Docker daemon:
failed to initialize logging driver: journald is not enabled on this host
To avoid this, you can override the Docker log driver by setting the
MORIO_DOCKER_LOG_DRIVER
environment variable. Setting it to local
should
do the trick:
export MORIO_DOCKER_LOG_DRIVER=local
npm run dev
Run Morio without network access
If you want to be able to work on Morio on the train, on a plane, at a cabin in the woods, or any other place where you do not have a network connection, you need to take some extra steps.
Pre-pull Docker images
The Morio core service will pull Docker images the first time they are used. If you already ran Morio, the images will be available locally. But if the first time you attempt to start Morio the network is not available, core won’t be able to pull the image, and Morio won’t start properly.
To avoid that, you can pre-pull all images by running:
npm run pull-oci
Provide a routable IP address
Morio needs a routable IP address to point to your machine. You cannot use
localhost
or a loopback IP address like 127.0.0.1
because inside a
containers these will resolve to the container’s loopback interface, rather
than your machine’s interface.
When network is available, you can use a service like nio.io
as we covered
above. Alas without network available we need to
help ourselves.
To do that, you can assign a routable IP address as an alias to your
loopback interface. For example, we can create an alias using IP address
10.10.10.10
:
sudo ifconfig lo0 alias 10.10.10.10 netmask 255.255.255.0
To remove the alias again, run this command:
sudo ifconfig lo0 -alias 10.10.10.10
Provide name resolution for our IP address
With a routable IP address in place, we still need to ensure it resolves correctly.
On your machine
Add a line to /etc/hosts
for the DNS name you want to use for your local
Morio setup. If we use local.morio.it
as an example along with our
10.10.10.10
example IP address, you need to add a line like this:
10.10.10.10 local.morio.it
Now your own machine will resolve local.morio.it
to 10.10.10.10
, even
without network access.
Inside the containers
Editing the /etc/hosts
file on our machine does not mean that inside the
containers local.morio.it
will also resolve 10.10.10.10
. That will only
work on our own machine.
To fix that, you can tell Morio to use ensure a custom name resolves to the
IP of your choice by setting the MORIO_DOCKER_ADD_HOST
environment variable
to a fqdn:ip
value:
export MORIO_DOCKER_ADD_HOST="local.morio.it:192.168.1.4"
With that, you can run Morio without the need for network access.
Run scripts
The repository uses a number of NPM run scripts. These are scripts that help with a variety of housekeeping and automation tests and can be run by issuing the following command in the root:
npm run name
where name
is the name of the run script.
We will cover the main run scripts here. For a full list and all details, refer to the run scripts reference documentation for contributors.
npm run kickstart
We’ve already mentioned it when we explained how to get started, but this run script will install dependencies and set up the monorepo for you.
To use this, run the following command in the root:
npm run kickstart
npm run build
This will build development versions of the various container images required by Morio to run the local development environment.
To use this, run the following command in the root:
npm run build
npm run eslint
This will run the ESLint JavaScript linter on the .
The linter will find potential issues in your code that you should fix. The linter will also run in when submitting a pull request, so running it locally first means you can be confident that your pull request will pass that test.
To use this, run the following command in the root:
npm run eslint
npm run prettier
This will run the Prettier JavaScript code front matter on the .
The front matter will first reformat your code according to our coding standards. The front matter will also run in when submitting a pull request, so running it locally first means you can be confident that your pull request will pass that test.
To use this, run the following command in the root:
npm run prettier
npm run dev
This will run the local development environment. In other words, it will start Morio locally.
To run this, you should first run npm run build
. Afterwards, you do not
need to rebuild again unless the version changes. Your local changes will
be picked up automatically, and the relevant services will be reloaded.
What will happen exactly depends a bit on the various services, but in general, starting Morio this way will run the code in your local copy of the rather than any pre-built code.
To use this, run the following command in the root:
npm run dev
npm run logs:core
This will tail the logs of the core service and output them in a nicely formatted way. As logs are formatted in JSON, this will make them easier to read while developing.
To use this, run the following command in the root:
npm run logs:core
This requires the pino-pretty NPM package to be installed. To install it, run:
npm install --global pino-pretty
npm run logs:api
This will tail the logs of the api service and output them in a nicely formatted way. As logs are formatted in JSON, this will make them easier to read while developing.
To use this, run the following command in the root:
npm run logs:api
This requires the pino-pretty NPM package to be installed. To install it, run:
npm install --global pino-pretty
npm run destroy
This will destroy the local development environment. In other words, it will
undo what npm run dev
did.
To use this, run the following command in the root:
npm run destroy
You will not lose any code changes you made.
npm run redev
This will destroy the local development environment, and create a fresh one.
It combines npm run destroy
with npm run dev
.
npm run redev
You will not lose any code changes you made.
npm run test:api
This will run the unit tests on the api service.
To use this, run the following command in the root:
npm run test:api
This will destroy your local development environment, although it will set it up again after the tests have completed.
You will not lose any code changes you made.
npm run test:core
This will run the unit tests on the core service.
To use this, run the following command in the root:
npm run test:core
This will destroy your local development environment, although it will set it up again after the tests have completed.
You will not lose any code changes you made.
npm run pull-oci
This will pull all Docker images used by Morio. This prevents from core having to do so when you set up Morio.
To use this, run the following command in the root:
npm run pull-oci
npm run help
This will output information on where to get help with/for Morio.
To use this, run the following command in the root:
npm run help
When you get stuck, this will tell you where to get help.
About our use of GitHub
We use GitHub for hosting our because that’s where the largest community of potential contributors are. But we also maintain a mirror at code.europa.eu/certeu/morio (a GitLab instance) and in general like to keep our options open and not be beholden to any particular code hosting platform.
The area where that goal of flexibility requires the most discipline is perhaps . In general, CI can refer to any sort of automation that is triggered by changes in the code base, such as a commit or a pull request. However, in the context of GitHub, many people equate CI with GitHub Actions, GitHub’s own flavour of CI that has grown into an ecosystem.
The problem with this is that we want whatever runs in CI to be available locally. A faster feedback loop means you can iterate faster. Nobody should have to commit, push, and then wait for CI to complete to know whether a given pipeline passes.
While there are solutions to run GitHub Actions locally, they add a non-trivial amount of complexity. Better is to avoid these pitfalls by creating pipelines that run locally first, and then calling them from CI.
To ensure this, please consider the following choices when considering CI or automation:
- Can it run locally?
- Does it make sense to run this locally?
Only if the answers are No and No again should you consider a GitHub action that only runs on GitHub. For all other cases, please create a local run script.
The decision tree below visualises these rules: