Configuration
The configuration file marv.conf
is in Python config parser / ini-syntax and consists of at least one marv section and one or more collection section. See some Examples below.
If you make changes to your configuration, keep in mind that you have to stop gunicorn
, run marv init
, and start gunicorn
again.
Relative paths
The location of marv.conf
is the site directory and relative paths are relative to that directory.
marv section
[marv]
ce_anonymous_readonly_access (CE)
Give anonymous users readonly access to all datasets.
Example:
ce_anonymous_readonly_access = True
collections
Name of one or more collections, corresponding to a collection section.
Example:
collections = bags
dburi
Location of sqlite database. Despite the generic name, only sqlite is supported.
Example:
dburi = sqlite:////var/local/lib/marv/db/db.sqlite
Default:
dburi = sqlite:///path/to/sitedir/db/db.sqlite
Note
Keep the db/db.sqlite
suffix for ease of migrations.
reverse_proxy
When marv is running behind a reverse proxy, serving of files can be offloaded for greatly improved performance. Currently, the only supported reverse proxy is nginx
.
Example:
reverse_proxy = nginx
See Gunicorn behind NGINX for the corresponding nginx configuration.
leavesdir
Storage location for datasets uploaded by leaves.
Example:
leavesdir = /var/local/lib/marv/leaves
Default:
leavesdir = ./leaves
oauth (EE)
See OAuth2 (EE).
oauth_enforce_username
In case of exactly one oauth (EE) provider, take username from oauth response key, instead of letting the user choose herself. See OAuth2 (EE) for more information.
oauth_gitlab_groups
Require GitLab oauth users to be a member of at least one of a comma-separated list of groups. See OAuth2 (EE) for more information.
smtp_from
Sender address MARV shall use when generating emails.
Example:
smtp_from = marv@example.com
Default:
smtp_from =
smtp_url
Server host, port and credentials for sending emails from MARV. STARTTLS will be used and smtp://
is the only supported schema.
Example:
smtp_url = smtp://marv_user:marv_password@mail.example.com:587
smtp_url = secret://smtp_url
The second example will read the url from a top-level smtp_url
key from site/secrets.json
, see secrets (EE) for more information.
Default:
smtp_url =
storedir
Example:
storedir = /var/local/lib/marv/store
Default:
storedir = ./store
collection section
Configuration for a collection of datasets.
scanner
A scanner is responsible to group files into named datasets.
Example:
scanner = marv_robotics.bag:scan
scanroots
One or more directories to scan for datasets.
Example:
scanroots =
./foo
./bar
Warning
MARV Robotics does not need write access to your bag files. As a safety measure install and run MARV as a user having only read-only access to your bag files.
nodes
List of nodes made available within this collection under the name following the column, which is also the name of the function the node is created from. When listing colums or filters are added, the given extractor function run for all the collection’s datasets. For this to be quick, all node output used in listing colums and filters must be readily available. Therefore all nodes listed in the configuration are persisted in the store. For a node to be persisted it needs to define a message schema. See Declare image node for an example.
Example:
nodes =
# pkg.module:func_name
marv_nodes:dataset
marv_robotics.bag:bagmeta
For a list of nodes see Nodes.
filters
Listings of datasets for the web frontend and API responses can be filtered.
Nodes extract and process data from datasets. Node output persisted in the store is available via a node’s name. For this to happen the node needs to define a message type and be listed in nodes. Filters and listing_columns use S-Expressions to extract values from node output. Via API or web frontend a user supplies filter input to be compared with the extracted value using a selected operator. A filter’s name is displayed in the web frontend and its ID is used via API.
At least one operator has to be configured per filter and valid operators depend on the field type. The tags
and comments
filter are special and have to be defined exactly as shown below.
See S-Expressions on how to create functions to extract values from node output.
Example:
filters =
# id | Display Name | operators | field type | extractor function
name | Name | substring | string | (get "dataset.name")
setid | Set Id | startswith | string | (get "dataset.id")
size | Size | lt le eq ne ge gt | filesize | (sum (get "dataset.files[:].size"))
tags | Tags | any all | subset | (tags )
comments | Comments | substring | string | (comments )
fulltext | Fulltext | words | words | (get "fulltext.words")
files | File paths | substring_any | string[] | (get "dataset.files[:].path")
end_time | End time | lt le eq ne ge gt | datetime | (get "bagmeta.end_time")
duration | Duration | lt le eq ne ge gt | timedelta | (get "bagmeta.duration")
topics | Topics | any all | subset | (get "bagmeta.topics[:].name")
In case you use emacs, it’s easy to align these: C-u M-x align-regexp | RET RET y
.
field type
The field type determines what python type the extractor function is expected to return, how this is interpreted and displayed, and what is expected as filter input.
datetime
lt
le
eq
ne
ge
gt
filesize
lt
le
eq
ne
ge
gt
float
lt
le
eq
ne
ge
gt
int
lt
le
eq
ne
ge
gt
string
substring
, startswith
string[]
substring_any
subset
any
, all
timedelta
lt
le
eq
ne
ge
gt
words
words
operators
lt
le
eq
ne
ge
gt
Comparison of numeric input with numeric stored value.
substring
Match input as substring anywhere in stored string.
startswith
Stored string starts with input string.
substring_any
The input string is a substring of any string in a stored list of strings.
any
The set of input strings intersects with the set of stored strings.
all
The set of input strings is a subset of the set of stored strings.
listing_columns
Columns displayed for the collection’s listing.
For certain colums the id is important, so keep the ids used in the Default configuration. The heading is used as column heading, formatters are explained below and see S-Expressions on how to write functions to extract values from node output.
Example:
listing_columns =
# id | Heading | formatter | extractor function
name | Name | route | (detail_route (get "dataset.id") (get "dataset.name"))
size | Size | filesize | (sum (get "dataset.files[:].size"))
formatter
Marv ships with a set of formatters. See Custom on how to override these and supply your own.
acceleration
Renders numeric value with unit. Unit can be chosen in frontend (EE).
date
datetime
distance
Renders numeric value with unit. Unit can be chosen in frontend (EE).
float
Renders float with two decimal places.
icon
Render a glyphicon by name (glyphicon-<name>
) with optional additional space-separated css classes and a title rendered in a tooltip for the icon.
{'icon': name, 'classes': css_classes, 'title': title}
int
link
{'href': '', 'title': ''}
pill
route
Used only for the detail route so far in conjunction with detail_route.
speed
Renders numeric value with unit. Unit can be chosen in frontend (EE).
string
timedelta
listing_sort
Column and sort order for listing.
Example:
listing_sort = start_time | descending
The first field corresponds to an id in listing_columns, the second is one of ascending
(default) or descending
.
listing_summary
Summary calculated for the filtered rows of the listing.
Example:
listing_summary =
# id | Title | formatter | extractor
datasets | datasets | int | (len (rows))
size | size | filesize | (sum (rows "size" 0))
duration | duration | timedelta | (sum (rows "duration" 0))
A unique id, a title displayed below the value, a formatter explained in formatter and extractor function explained in S-Expressions.
detail_summary_widgets
List of widgets to be rendered on the first tab of the detail view, aka the summary section.
Example:
detail_summary_widgets =
summary_keyval
bagmeta_table
You can write your own and use some of the already existing Widget nodes.
detail_sections
List of detail section to be rendered beyond the summary section. For any given dataset those sections will be rendered only if the dataset contains the necessary data. In absence of meaningful data, sections will be omitted from the web frontend detail view.
Example:
detail_sections =
connections_section
video_section
You can write your own and use some of the already existing Section nodes.
secrets (EE)
Web-based user, group, and leaf management (EE) and OAuth2 (EE) require secrets. Instead of storing these in marv.conf
store them in secrets.json
in the site directory.
{
"smtp_url": "smtp://marv_user:marv_password@mail.example.com:587"
}
Secrets are referenced from marv.conf
by using secret://<key name>
, see smtp_url for an example.
S-Expressions
S-Expressions are used in the config file to create small functions that extract values from output of stored nodes. S-Expressions are (nested) lists in parentheses, with list elements being separated by spaces.
(get "dataset.name")
(get "dataset.files[:].path")
(sum (get "dataset.files[:].size"))
(tags)
(comments)
The first element of a list is the name of a function. Any additional arguments are passed as arguments to the function and the list defining a function is replaced with its return value.
Valid arguments are:
functions enclosed in
()
all json literals -
null
-true
,false
- integers-17
,42
, … - floats1.2
,-1e10
-"strings with escape sequences \u0022 \" \\ \b \f \n \r \t"
Functions
Functions in S-expressions get and process data from store node output. Some may be used in all scopes filters, listing_columns, and listing_summary; some only in some (see below).
detail_route
Return dictionary rendering link to detail route of dataset. First argument is the dataset’s setid, second optional name is displayed instead of setid.
scope: listing_columns
filter
Filter list by removing null
elements (Python None
).
Examples:
(filter null (makelist (get "unit.name")))
scope: filters, listing_columns
format
Wrapper for fmt.format(*args)
. First argument is the format string fmt
, remaining arguments are passed on.
scope: filters, listing_columns
get
Get a value from a nodes output. First argument defines node and traversal into its output, second optional argument is used as default value instead of None
.
Examples:
(get "bagmeta.start_time")
(get "dataset.files[:].size")
The specifier starts with the nodes name. A .
performs dictionary key lookup. Lists can be traversed into in part or full using slicing and further dictionary lookup is performed on each element of the list.
scope: filters, listing_columns
getitem
Get an item from a list or dictionary.
(getitem (split (get "foo.bar") "/" 1) 0)
scope: filters, listing_columns
join
Wrapper for joinstr.join(args)
. First argument is the join string remaining arguments are joined with.
scope: filters, listing_columns
len
Return length of first argument.
scope: filters, listing_columns, listing_summary
link
Render link with first argument as href and second argument displayed.
scope: listing_columns
makelist
Takes one or more arguments and returns a list containing these.
Examples:
(filter null (makelist (get "unit.name")))
scope: filters, listing_columns
max
Return maximum element of first argument.
scope: filters, listing_columns, listing_summary
min
Return minimum element of first argument.
scope: filters, listing_columns, listing_summary
rows
Return all rows matching current filter criteria. The optional second and third arguments extracts a specific column defined in listing_columns instead of the full row and provide a default value for it.
Examples:
(sum (rows "size" 0))
scope: listing_summary
rsplit
Split string from the right. First argument is the string, further arguments are passed to python’s string rsplit method.
scope: filters, listing_columns
set
Return set with items from one iterable argument.
scope: filters, listing_columns, listing_summary
split
Split string. First argument is the string, further arguments are passed to python’s string split method.
scope: filters, listing_columns
sum
Return sum of arguments.
scope: filters, listing_columns, listing_summary
trace
Print trace messages.
scope: filters, listing_columns
Examples
Default configuration
[marv]
collections = bags
# Use next line to run behind nginx
# reverse_proxy = nginx
[collection bags]
scanner = marv_robotics.bag:scan
scanroots =
/scanroot
nodes =
marv_nodes:dataset
marv_robotics.bag:bagmeta
marv_robotics.cam:ffmpeg
marv_robotics.cam:images
marv_robotics.detail:bagmeta_table
marv_robotics.detail:connections_section
marv_robotics.detail:gnss_section
marv_robotics.detail:images_section
marv_robotics.detail:summary_keyval
marv_robotics.detail:trajectory_section
marv_robotics.detail:video_section
# marv_robotics.fulltext:fulltext
marv_robotics.gnss:gnss_plots
marv_robotics.motion:acceleration
marv_robotics.motion:distance_gps
marv_robotics.motion:motion_section
marv_robotics.motion:speed
marv_robotics.trajectory:trajectory
filters =
# id | Display Name | operators | value type | value function
name | Name | substring | string | (get "dataset.name")
setid | Set Id | startswith | string | (get "dataset.id")
size | Size | lt le eq ne ge gt | filesize | (sum (get "dataset.files[:].size"))
status | Status | any all | subset | (status)
tags | Tags | any all | subset | (tags)
comments | Comments | substring | string | (comments)
# fulltext | Fulltext | words | words | (get "fulltext.words")
files | File paths | substring_any | string[] | (get "dataset.files[:].path")
added_time | Added | lt le eq ne ge gt | datetime | (get "dataset.time_added")
start_time | Start time | lt le eq ne ge gt | datetime | (get "bagmeta.start_time")
end_time | End time | lt le eq ne ge gt | datetime | (get "bagmeta.end_time")
duration | Duration | lt le eq ne ge gt | timedelta | (get "bagmeta.duration")
topics | Topics | any all | subset | (get "bagmeta.topics")
msg_types | Message types | any all | subset | (get "bagmeta.msg_types")
listing_columns =
# id | Heading | formatter | value function
name | Name | route | (detail_route (get "dataset.id") (get "dataset.name"))
size | Size | filesize | (sum (get "dataset.files[:].size"))
tags | Tags | pill[] | (tags)
added | Added | datetime | (get "dataset.time_added")
start_time | Start time | datetime | (get "bagmeta.start_time")
duration | Duration | timedelta | (get "bagmeta.duration")
max_speed | Max speed | speed | (max (get "speed[:].value"))
distance | Distance | distance | (sum (get "distance_gps[:].value"))
listing_sort = start_time | descending
listing_summary =
# id | Title | formatter | extractor
datasets | datasets | int | (len (rows))
size | size | filesize | (sum (rows "size" 0))
duration | duration | timedelta | (sum (rows "duration" 0))
distance | distance | distance | (sum (rows "distance" 0))
detail_summary_widgets =
summary_keyval
bagmeta_table
detail_sections =
connections_section
images_section
video_section
gnss_section
motion_section
trajectory_section
System-wide configuration
/etc/marv/marv.conf
[marv]
collections = bags
# keep db/db.sqlite as the suffix!
dburi = sqlite:///var/local/lib/marv/db/db.sqlite
# store could also be somewhere else
store = /var/local/lib/marv/store
...
Multiple collections
[marv]
collections = bags bags2 videos
[collection bags]
scanner = marv_robotics.bag:scan
...
[collection bags2]
scanner = marv_robotics.bag:scan
...
[collection videos]
scanner = my_own_scanner:scan
...
comments
Return list of unicode objects with text of comments.
scope: filters, listing_columns