New custom blocks for Analytics Builder (async comms, downsampling and complex measurements)
Overview
We have updated the non-productized blocks in the open-source repository:
GitHub - SoftwareAG/analytics-builder-blocks-contrib: Unsupported, not...
Unsupported, not productized blocks for use with Apama Analytics Builder - SoftwareAG/analytics-builder-blocks-contrib
The repository contains blocks that you can deploy into Analytics Builder to enhance its functionality or to use as the basis for your own custom block development. This article introduces the following custom blocks:
- Async Send & Receive blocks to send signals inside and across models
- Time series Downsample block to derive a lower frequency time series from a high-frequency one
- Complex Measurement output block to create measurements containing multiple series
Async Signal - Go To (not) considered harmful
Analytics Builder does not directly support loops. This was a conscious design decision to avoid the creation of overly complex models that are hard to understand. For use cases requiring complex control logic, using EPL Apps is recommended.
Still, there are cases, where routing back to earlier blocks in a model is necessary. A typical case is resetting the state of block(s) based on some condition identified later in the model. For example, you are calculating the average of a measurement and want to reset the average for every 10 received measurements. One way of doing that is like this:
But now we are calculating the count twice: once in the Counter block and once in the Discrete Statistics block. That may be ok here, but if the decision when to reset is more complex, replicating it multiple times can be very complex.
Here the Async Signal blocks can help: the Send Async Signal sends an event and the Receive Async Signal block can receive it. This allows to not only send a signal to an earlier block in the same Analytics Builder model but also to other models.
In our case, the model would look like this:
Notice how every time the count from the Discrete Statistics block is larger than 10, a signal named Reset is sent and how it is received by a block connected to the reset input of the Discrete Statistics block.
Arbitrary names can be used for the signal as long as they match across the sending and receiving blocks and are unique in the scope in which they are used.
Here you can see how the two blocks can be parameterized:
- Signal Type - the name or type of the signal. Must match the sender and receiver and needs to be unique within the used scope.
- Scope to Model - enabled by default. Restricts the signal to the current Analytics Builder model. To avoid unexpected behavior, it is recommended to keep enabled unless cross-model signalling is required.
- Scope to Partition - enabled by default. Restricts the signal to the current partition of the model. Mostly relevant for models running for device groups. In most cases, this should be kept enabled unless signals need to be sent from one device to all other devices in the group.
In addition to sending signals, arbitrary data can also be sent from the sender to the receiver block. For this purpose, the Send Async Signal block has a properties input. Any data provided to this input using, e.g., the Set Properties block, is available as properties on the output of the Receive Async Signal block and can be accessed, e.g., using the Extract Property block.
Attention: These asynchronous signaling blocks circumvent the loop detection of Analytics Builder. It is very easy to produce infinite loops inside or across models that produce large amounts of data in a short period of time. Carefully check any models using them before activating the models.
Time series Downsample
High-frequency time series are useful to analyze the behavior of devices and detect issues but storing them for long periods is costly and can have a negative impact on query performance. The latter is especially problematic when visualizing time series on a graph as it affects user experience. To make matters worse, the result may not be very appealing since too many data points can make charts difficult to understand.
Time series downsampling can help here. The Cumulocity IoT API out of the box supports downsampling but only provides a minimum and a maximum per downsampled period and hides when in the time period these happened.
There are many different algorithms that can perform time series downsampling, we provide a very simple version here. For every downsampled period, four data points are kept including the time when they happened: first, minimum, maximum and last data point.
The block looks like this:
It takes a value input which is the time series and has a sample input. Every time the sample input receives a pulse a downsampling period is being finished. In most cases, the sample input would be connected to a block that triggers it in regular intervals but this is not strictly necessary. At the end of a downsampling period, the block produces an output of downsampled measurements. These are provided via a pulse output that contains a measurements property which is a dictionary containing the four values. The keys of the dictionary are the timestamps when the values occurred.
As this output cannot be used with the standard Measurement Output block an additional block for Batch Measurements Output is provided. In contrast to the standard block, it does not have a Value or Properties input but only a Send input. This expects a pulse signal containing a measurement property as described above.
Together the usage of the two blocks in a model looks like this:
Note: The downsampled time series will be stored in addition to the original time series thus increasing storage costs. If the original time series is no longer required either provide a shorter retention period or use a processing mode that prevents storage of the original time series.
Complex Measurements
The Measurement Output block of Analytics Builder creates measurements with a single series. If an Analytics Builder model contains multiple Measurement Output blocks each produces a separate measurement even if they get activated at the same timestamp.
This makes sense if series are independent but not if they belong together, e.g., the x, y, and z components of an acceleration event.
The Complex Measurement Output block supports such use cases. Instead of accepting a single floating point value input, it accepts a pulse where the individual measurement values are provided as properties with fragment.series keys. Additionally, the block takes only the measurement type and not the fragment and serves as a parameter:
Imagine you receive c8y_Accleration measurements and want to calculate the delta between subsequent measurements and write the result as a new measurement, the model to achieve that could look like this:
The delta is calculated separately for the x, y, and z components and then written to properties using the Set Properties block. The result is sent to the *Complex Measurement Output block to create a single measurement containing three series.
In this example, the properties are only the floats coming out of the Delta blocks but they can also be objects containing the float as a value property and an additional unit property.
Note: The standard Cumulocity IoT Measurement input block analyzes the chains between models to avoid loops using the configured fragment & series. As these are dynamic with the Complex Measurement Output block, such analysis is not possible and you have to make sure that no loops exist yourself.
Summary
In addition to using the Block SDK to build and upload the blocks from your local machine, you can also use the Cumulocity IoT Analytics Management extension to directly deploy the blocks from inside your Cumulocity IoT tenant.