Skip to main content

Tap Stream Processors

Stream processors for the tap service are written in Javascript, and will be automatically bundled by Morio core.

The stream processor object​

Each processor is bundled a so-called stream processor object or . This SPO has the following properties:

id​

Mandatory

The id property of an holds a string that is the ID of the stream processor.

processor.id
const processor = { 
  id: `my_custom_stream_processor_with_a_unique_id`,
}
A processor ID is (typically) unique

The stream processor ID is used to bundle the processor code. In general, you should ensure the ID is unique. However, when two stream processors share the same ID, the later-loaded on will overwrite the earlier-loaded one.

So while in general, stream processor IDs should be unique, you can deliberately use this to overwrite a stream processor in the generated bundle.

info​

Recommended

The info property of the holds a string that is a description of the stream processor. It is intended for humans to clarify the role and purpose of the stream processor.

processor.info
const processor = { 
  info: `This stream processor shows how it's done`,
}

settings​

Mandatory

The settings property of the holds the stream processor’s configuration.

settings.topics​

Mandatory

The settings.topics property of an holds an array of strings that are the names of the Kafka topics to subscribe to.

processor.settings.topics
const processor = { 
  settings: {
    topics: ['logs'],
  }
}

settings.modules​

Optional

The settings.modules property of an holds an array of strings that are the names of the Morio modules to subscribe to.

processor.settings.modules
const processor = { 
  settings: {
    modules: ['linux-system'],
  }
}
Modules are optional

If no settings.modules are specified, the stream processor will be subscribed to (data generated by) all modules.

settings.datasets​

Optional

The settings.dataset property of an holds an array of strings that are the names of the Morio datasets to subscribe to.

processor.settings.datasets
const processor = { 
  settings: {
    datasets: ['journald'],
  }
}
Datasets are optional

If no settings.datasets are specified, the stream processor will be subscribed to (data generated by) all datasets.

settings.custom​

Optional

The settings property can also hold custom keys that allow you to expose the configuration of the underlying stream processor.

How they are configured depends on the data type they hold:

note

The data type below is used to render these settings in the Morio UI. It is not enforced by Morio core or the Tap service.

Custom settings with a number value​

Use dflt to specify the default value, and title to descibe the setting. Make sure to set type to number:

Example of a custom setting that holds a number
const processor = { 
  settings: {
    some_number: {
      dflt: 666,
      title: `Just an example of a number you can configure`,
      type: 'number'
    }
  }
}
Custom settings with a string value​

Use dflt to specify the default value, and title to descibe the setting. Make sure to set type to string:

Example of a custom setting that holds a string
const processor = { 
  settings: {
    some_string: {
      dflt: 'hello there',
      title: `Just an example of a string you can configure`,
      type: 'string'
    }
  }
}
Custom settings with a list of options​

Use dflt to specify the default value, and populate list with objects to describe the choices, where each holds:

  • val: the value
  • label: the lable
  • about: Optional extra text to clarify the choice

Make sure to set type to list:

Example of a custom setting that holds a string
const processor = { 
  settings: {
    my_list: {
      dflt: true,
      title: `Do you like Morio?`,
      type: 'list',
      list: [
        {
          val: false,
          label: `I don't even like myself`,
        },
        {
          val: true,
          label: `I love it`,
          about: 'Ok, maybe love is a bit much, so choose this option if you think it's ok.'
        },
      ],
    },
  }
}

handler​

Mandatory

The handler property of a stream processor holds a Javascript function that implements the logic of your stream processor.

processor.handler
const processor = { 
  handler: function(params) {
    // do something here
  },
  // Rest of the stream processor object
}

The handler function will be invoked for every message that matches the subscription data (topics, modules, and datasets).

tip

Refer to Tap params for details about what your handler function receives as parameters.

Refer to the Tap Stream Processing Guide for help on how to write your own logic.

Folder structure​

Stream processors are seeded in Morio. In other words, they are not part of the configuration, and are expected to be loaded from a git repository. You can use your own git repository for your custom stream processors, refer to the seeding guide for details.

For the seeding to function, you should make sure to respect the following rules:

  • Morio core will recursively look for index.mjs files
  • It will import the default export from those files
  • If the default export is an array, it will be treated as an array of stream processor objects
  • If the default export is an object, it will be treated as a single stream processor object

File structure​

A single object as default export​

The default export can be a single object like this:

Single object as default export
const processor = { 
  id: `my_custom_stream_processor_with_a_unique_id`,
  // Rest of the stream processor object
}

export default processor

An array as default export​

You can bundle multiple stream processors in the same file by exporting an array:

Single object as default export
const processors = [
  { 
    id: `my_first_custom_stream_processor_with_a_unique_id`,
    // Rest of the first stream processor object
  },
  { 
    id: `my_second_custom_stream_processor_with_a_unique_id`,
    // Rest of the second stream processor object
  },
]

export default processors

Exporting via a barrel file​

If you have a bunch of processors, a common approach is to keep each in its own file, and make the index.mjs a barrel file that merely re-exports them:

Using a barrel file
import processor1 from './my-great-processor.mjs'
import processor2 from './my-even-better-processors.mjs'
import otherProcessor from './this-is-a-test.mjs'

export default [
  processor1,
  processor2,
  otherProcessor
]

By naming your barrel file index.mjs Morio core will pick it up, and follow the imports. This gives you maximum flexibility on how to structure your code.