The source code for Sparv's graphical user interface is available at GitHub. It is divided into three parts:
- backend: a Flask application, providing the REST-API
- catapult: an auxiliary tool for running the Sparv pipeline
The code is distributed under the MIT license. Please refer to the above repositories for installation instructions.
The Sparv pipeline is also required for running the Sparv interface.
Technical Documentation of the frontend
This section describes the internal structure of the Sparv frontend and can be used as a developer's guide.
The frontend is available at https://spraakbanken.gu.se/sparv/.
- Node v0.10.x, which should be installed through you package manager. This should by default include NPM, the node package manager.
- Grunt, install using
npm install -g grunt-cli.
- CoffeeScript, install using
npm install -g coffeescript.
- Sass, install using
npm install sass.
app/config.js file contains the configuration of the backend address, the address to the default settings JSON schema and also the address to Karp.
Running the Frontend
For running the frontend locally (while developing) run
grunt serve. In you browser, open
http://localhost:9010 to launch Sparv. While running
grunt serve the CoffeeScript and Sass files are automatically compiled upon edit, additionally causing the browser window to be reloaded to reflect the new changes.
Before releasing a new version, the scripts are compiled by running
grunt in the frontend directory. This will create a directory called
dist which contains all the files necessary to run the frontend.
Technical Documentation of the backend
This section describes the internal structure of the Sparv backend and its usage of the catapult and can be used as a developer's guide.
The backend is run through the WSGI script
index.py. It is available at https://spraakbanken.gu.se/ws/sparv.
- Version 3 of the Sparv corpus pipeline (see the technical report for installation instructions)
- Python 3.4 or newer
Python virtual environment
Though it is not required, we recommend that you use a Python virtual environment to run the Sparv backend. This is the easiest way to ensure that you have all the Python dependencies needed to run the modules.
Set up a Python virtual environment as a subdirectory to the
python3 -m venv venv
Activate the virtual environment and install the required Python packages:
source venv/bin/activate pip install -r requirements.txt
You can then deactivate the virtual environment:
The configuration variables are stored in
backend: the address where the backend is hosted
sparv_python: the Python path to the sparv pipeline python directory
sparv_backend: the path to the backend directory
builds_dir: the directory that hosts running and completed builds
log_dir: location of the log files. Can be set to
Noneto log to stdout.
sparv_models: location of the sparv pipeline models
sparv_makefiles: location of the sparv pipeline makefiles
secret_key: a string of your choosing. It is needed for queries that may cause deletions of builds.
venv_path: path to the activation script of the Python virtual environment (may be set to None)
processes: number of processes that
makewill run while annotating
fileupload_ext: extension used for builds that contain file uploads
socket_file: path to the socket file used to communicate with the catapult
catalaunch_binary: path to the catalaunch binary file
python_interpreter: "Python" interpreter, replaced with catalaunch
When running the backend with gunicorn (recommended) you may also want to modify the configuration in
The suggested location for the Sparv pipeline is the directory
Running the Backend
The backend is set up to be run with gunicorn which is installed automatically inside the Python virtual environment. From the backend directory you can run the following command:
html/app/venv/bin/gunicorn -c html/app/gunicorn_config.py index
This will start a WSGI server and bind it to the socket defined in
gunicorn_config.py. Log messages are written to the file specified in the same file (or to the terminal if nothing is specified).
The backend can also be started by running
index.py with the Python interpreter but this is mostly used for development or debugging.
Makefile and Settings JSON Schema
The makefile for each corpus is created from a JSON object that is created by the script
html/app/schema_generator.py. The frontend builds its form based on the requested schema. New entries can be added and the frontend should render them automatically. The file that creates the makefile is
The catapult runs a Python instance that shares the loaded lexicons, keeps malt processes running and lowers the Python interpreter startup time. Scripts are run on the catapult with the tiny c program
The catapult can for example be placed inside the
Just as for the Sparv backend we recomment that you use a Python virtual environment for running the catapult. Check the backend requirements for instructions. The standard location for installing the virtual environment for the catapult is inside the
After setting up the Python virtual environment you need to adapt the variable
VENV_PATH in the Sparv pipeline in
/makefiles/Makefile.config so it points to the catapult virtual environment.
The configuration variables are stored in
SPARV_PYTHON: the path to the
sbpipeline Python directory
SPARV_MODELS: location of the
SPARV_BIN: location of the
SPARV_MAKEFILES: location of the pipeline makefiles
CATAPULT_DIR: location of the catapult directory
BUILDS_DIR: the directory that hosts running and completed builds
LOGDIR: the path to the log file directory
CATAPULT_VENV: the path to the Python virtual environment used by the catapult
Running the Catapult
./start-server.shto start the catapult.
- Set up the cron jobs listed in
catapult/cronjobs. for the automatic maintenance of Sparv.
The following cron jobs are used in Sparv:
- Cleanup: Builds that have not been accessed for 7 days are removed every midnight by issuing
- Keep-alive: The script
catapult/keep-alive.shis run every five minutes and restarts the catapult with
catapult/start-server.shif it does not respond to ping. Instead of setting up this cron job you can run the catapult using a process control system like supervisord.
- Update-saldo: The Saldo lexicon is updated daily with the script
catapult/update-saldo.sh. This takes some time, and is therefore run during the night. The catapult is restarted afterwards by the
Interaction between the Sparv components
The following image illustrates how the components involved in the Sparv web interface interact with each other and with the user, both before and during the analysis.