A small helper library to group common AIT RDP functions and to provide a unified user experience
The pyrdp-commons library is a loose collection of helper functions that are regularly used within the AIT RDP. Right now, the lib mostly assists in parsing an extended configuration format and setting up the execution environment. In addition, dynamic configuration that automatically fetches information from remote endpoints and receives configuration updates is supported.
The configuration format is a superset of YAML using custom tags to enable various functionalities. However, using plain
YAML is already sufficient for several applications. In case the pyrdp_commons.cli.setup_app(...) or
async_setup_app(...) functions are used to setup the application, the following base directives are supported.
version: 1 # Optional version indicator to check file compatibility. Per default, version 1 is assumed.
# The optional logging configuration that is passed on to the python logging library. See the python reference on
# logging.config.dictConfig (https://docs.python.org/3/library/logging.config.html#logging.config.dictConfig) for
# further information
logging:
loggers:
pyrdp_commons.test.logger:
level: CRITICAL
handlers: [ "console" ]
# Any application-specific content can be provided here.
In case some applications support table arguments, external table provided as CSV can be referenced as follows. Note that relative files are resolved according to the directory of the current YAML file (if available).
input table: !table/csv # Indicates that an external table (pandas.DataFrame) is to be loaded in the CSV format
path: "extensive-table.csv" # The mandatory path to the table
sep: "," # An optional field separator. Per default ";" is used
In contrast to plain YAML, pyrdp-commons supports templates that can be filled with external information. In all
strings, variable substitution is supported. In case the environment (or context) variable name is set to “AIT”, the
YAML string "Hello ${name}!" will be resolved to “Hello AIT!”. Dedicated template objects allow to repeat
configuration snippets according to some external, tabular information. For each row in the external data source, a
dedicated repetition is created. For each column, a variable will be defined and set.
data_sources:
"Static Source": # A static configuration that is directly adjacent to the dynamic one.
url: https://ait.ac.at
station_id: ATU 14703506
# A dynamic configuration that is repeated for each row in the source table.
# Note that the key will also be computed based on the table content. If there is a column called name, that name
# will be set for each repetition individually.
"Dynamic Source - ${name}": !template/repeat # Indicates that the entry is to be repeated
source table: !table/csv # Just loads the external table
path: "external-information.csv"
# The body content specifies the configuration structure that is repeated. In this case, for each source table row,
# a dict entry with the key "Dynamic Source - ${name}" and the dict value {"url": "${URL}", "station_id":
# "${Station_ID}"} (with the appropriate variables replaced) will be appended to data_sources.
body:
url: ${URL}
station_id: ${Station_ID}
In case the websrc extensions are installed, it is possible to fetch JSON-formatted content from remote REST services:
some source table: !table/rest # Load the data from a remote REST server
url: "http://127.0.0.1:8764/config" # The remote URL of the item
# Optional dictionary of parameters that will be appended to the URL. Although the parameters may be directly set in
# the URL field, the dict-version supports proper URL encoding of dynamically replaced variables.
parameters:
id: 42
# The authentication method. Right now, only OAuth2 using client credentials is supported. This field is mandatory.
auth:
type: "OAuth2-ClientToken" # Optional authentication type.
client_id: "ConfigClient" # The client ID as registered in the authentication backend
client_secret: "SuperSecret" # The client secret
# The token endpoint to fetch the actual authentication tokens that will be passed on to the actual web request.
token_endpoint: "http://127.0.0.1:8764/auth/token"
Per default, it is expected that the REST endpoint returns a list of dicts. Each dict corresponds to one table row and
each dict member to a named column. However, the column data can also be selected using JSONPath expressions and the
columns configuration directive:
some source table: !table/rest
url: "http://127.0.0.1:8764/config"
# The column dictionary defines which columns are returned. For each column, a JSONPath expression is expected that
# returns the data. In this case, two columns, content and name_info are extracted.
columns:
content: "$[*].nested.content"
name_info: "$[*].name"
auth:
type: "OAuth2-ClientToken"
client_id: "ConfigClient"
client_secret: "SuperSecret"
token_endpoint: "http://127.0.0.1:8764/auth/token"
In case the application supports dynamic configuration reloading, it is possible to amend templates by external event sources that trigger a reload operation. Currently, a Websockets-based protocol is supported and can be configured as follows:
main_container:
"${name}": !template/repeat
source table: !table/rest # Again a dynamic web source, but this can be any other source as well.
url: "http://127.0.0.1:8764/config"
auth:
type: "OAuth2-ClientToken"
client_id: "ConfigClient"
client_secret: "SuperSecret"
token_endpoint: "http://127.0.0.1:8764/auth/token"
# The "source events" parameter of the template specifies the event source that is used to dynamically reload the
# configuration.
source events: !events/websocket # Here, a Websocket-based protocol should be used.
url: "ws://127.0.0.1:8764/event?name=test-event" # The websocket URL to connect to.
auth: # The same authentication structure as for !table/rest
type: "OAuth2-ClientToken"
client_id: "ConfigClient"
client_secret: "SuperSecret"
token_endpoint: "http://127.0.0.1:8764/auth/token"
body: # Some dynamic content loaded from the remote source.
key: "${name}"
val: "${value}"
date: "${date}"
# Add the source repository
poetry source add gitlab-pyrdp-commons https://gitlab-intern.ait.ac.at/api/v4/projects/3611/packages/pypi/simple
# Setup the credentials. Don't use a username here. Just tokens are supported
poetry config http-basic.gitlab-pyrdp-commons __token__ ${TOKEN_PYRDP_COMMONS}
# Install the dependency including all extras
poetry add --source gitlab-pyrdp-commons pyrdp-commons -E websrc -E fullasync
Note that the following extras must be included to use the corresponding functionality:
websrc: Load the configuration and configuration events from remote REST and Websockets-based endpoints.fullasync: Use all asynchronous functions that may depend on external librariesIn case no event loop is running, the synchronous API is the way to go. As usual, the functions will block until all configurations are loaded. Note that there are high-level functions that setup the entire application. In addition, the YAML files can be loaded using some low-level API.
Most commonly, pyrdp-commons will be used to load the common configuration and setup the application (logging,
observability). Assume that the path to the source yaml file is given in config_file, the config will be read as
follows.
import pyrdp_commons.cli
config = pyrdp_commons.cli.setup_app(config_file)
It is also possible to parse any .env files, beforehand. In this case, a second parameter, env_file naming the
targeted file must be appended.
config = pyrdp_commons.cli.setup_app(config_file, env_file=".env")
Per default, the setup_app function returns standard python dict/list containers that naturally do not support the
advanced event mechanism. In case the extended containers should be used, the dst_type parameter needs to be set to
dst_type="extended".
If you directly and repeatedly need to load YAML files using the extended syntax, a pyrdp_commons.SyncConfigFactory is
provided that loads the corresponding configuration containers without configuring the application.
In case an event loop is already running, the synchronous YAML functions cannot be used. In this case, asynchronous
counterparts such as pyrdp_commons.cli.async_setup_app and an pyrdp_commons.AsyncConfigFactory for low-level access
are provided. Since the configuration update mechanism is currently also only available as asynchronous interface, the
async versions are the way to go, if configuration needs to be dynamically updated.
Each AbstractConfigContainer, which can either be a list-like (ConfigList) or dict-like (ConfigDict) container,
supports listening to configuration changes. Therefore, the root-level container needs to recursively watch for
configuration changes using its watch_and_fire() function. The asynchronous function will only return if no event
source is registered or all events are exhausted. Otherwise, it will block until the surrounding task is cancelled.
As soon as a configuration change is expected, the affected configuration templates will be reloaded and the event
notification mechanism is triggered. Each AbstractConfigContainer allows to register an event callback using the
set_on_change(...) function. In case no callback is registered or the callback function returns False, the event
will be further escalated to the parent container.
import asyncio
from pyrdp_commons import AsyncConfigFactory, AbstractConfigContainer, ChangeEvent
# The event listener that will receive configuration events. Can be both, a synchronous or asynchronous function.
async def handle_event(event: ChangeEvent):
print(f"Received a configuration update: {event}")
async def _main():
factory = AsyncConfigFactory()
config = await factory.create_config_from_yaml_file("config.yml") # Read the given config file
config.set_on_change(handle_event) # Register the event listener.
await config.watch_and_fire() # Watch for changes. Will block until all events are exhausted.
asyncio.run(_main())