Skip to content

How to make a processor

Jump to bottom
Stijn Peeters edited this page Jan 20, 2020 · 29 revisions

Processors

4CAT is a modular tool. Its modules come in two varietes: data sources and processors. This article covers the latter.

Processors are bits of code that produce a dataset. Typically, their input is another dataset. As such they can be used to analyse data; for example, a processor can take a csv file containing posts as input, count how many posts occur per month, and produce another csv file with the amount of posts per month (one month per row) as output.

4CAT has an API that can do most of the scaffolding around this for you so processors can be quite lightweight and mostly focus on the analysis while 4CAT's back-end takes care of the scheduling, determining where the output should go, et cetera.

Example

This is a minimal, annotated example of a 4CAT processor:

"""
A minimal example 4CAT processor
"""
from backend.abstract.processor import BasicProcessor

class ExampleProcessor(BasicProcessor):
	"""
	Example Processor
	"""
	type = "example-processor"  # job type ID
	category = "Examples" # category
	title = "A simple example"  # title displayed in UI
	description = "This doesn't do much"  # description displayed in UI
	extension = "csv"  # extension of result file, used internally and in UI

	input = "csv:body"
	output = "csv:value"

	def process(self):
		"""
		Saves a CSV file with one column ("value") and one row with a value ("Hello
		world") and marks the dataset as finished.
		"""
		data = {"value": "Hello world!"}
		self.write_csv_items_and_finish(data)

Processor properties

Processor settings and metadata is stored as a class property. The following properties are available and can be set:

  • type (string) - A unique identifier for this processor type. Used internally to queue jobs, store results, et cetera. Should be a string without any spaces in it.
  • category (string) - A category for this processor, used in the 4CAT web interface to group processors thematically. Displayed as-is in the web interface.
  • title (string) - Processor title, displayed as-is in the web interface.
  • description (string) - Description, displayed in the web interface. Markdown in the description will be parsed.
  • extension (string) - Extension for files produced by this processor, e.g. csv or json.
  • input (string) - Simple schema for the input files required by this processor, in the format type[:fields]. For example, csv:column1,column2 means the processor requires a CSV file with at least the columns column1 and column2 as input. The field definition is optional, i.e. just csv is also valid.
  • output (string) - Simple schema for the files produced by this processor.
  • option (dictionary) - A dictionary of configurable parameters for this processor. See the section on 'processor options' for more details.
  • interrupted (bool) - false by default. This may be set to true by 4CAT's scheduler. Once it is true, the processor should return as soon as possible, even if processing is not complete yet. If processing has not finished yet, the result so far (if any) should be discarded and the dataset status updated to reflect that it has been interrupted. Since abort procedures are different per processor, this is the processor's responsibility. The simplest way of addressing this is raising a ProcessorInterruptedException (found in backend.lib.exceptions); this will gracefully stop the processor and clean up so it may be attempted again later.

Processor API

The full API available to processors is as follows. All of these are members of the processor object, i.e. they are accessed via self.property within the processor code. While other methods or classes may be available within processors, relying on them is discouraged and unsupported; when possible, use only the API documented here.

Methods

  • process() -> void: This method should contain the processor's logic. Other methods may be defined, but this one will be called when an analysis with this processor is queued.
  • iterate_csv_items(pathlib.Path source_file) -> Generator: This yields one row of the provided CSV file per call and raises a ProcessorInterruptedException if the processor has been interrupted while iterating.
  • write_csv_items_and_finish(list data) -> void: Writes a list of dictionaries as a CSV as the result of this processor, and finishes processing. Raises a ProcessorInterruptedException if the processor has been interrupted while writing.

Objects

  • dataset (DataSet): The dataset produced by this processor
    • dataset.finish(int num_items_in_result) -> void: Manually mark the dataset as finished. Either this method or write_csv_and_finish() should always be called at the end of process().
    • dataset.get_result_path() -> pathlib.Path: Returns a Path object referencing the location the processor result should be saved to.
    • dataset.update_status() -> void: Updates the dataset status. This can be used to indicate processor progress to the user through the web interface. Note that updates the database, which is relatively expensive, and you should not call it too often (for example, not every iteration of a loop, but only every 500 iterations).
  • source_file (pathlib.Path): The file to be processed as input
  • parent (DataSet): The dataset used as input by this processor
  • parameters (dict): Options set by the user for this processor

Processor options

It may be useful to offer a user options through which they can configure a processor, e.g. to set the language to use for text analysis or the resolution of a visualisation. This is possible with the options property of the processor class. This option, if set, should be a dictionary, each item having its option identifier as a key and the option definition as a value, e.g.:

def SomeProcessor(BasicProcessor):
    options = {
        "graph-title": {
            "type": UserInput.OPTION_TEXT,
            "default": "Default title"
        },
        "graph-height: {
            "type": UserInput.OPTION_TEXT,
            "default": 25,
            "min": 0,
            "max": 100
        }

This would, in the web interface, show these two options to a user. The value set for these options, when running the processor, may then be accessed as follows:

    def process():
        height = self.parameters.get("graph-height", 10)
        # or, to use the default set before...
        height = self.parameters.get("graph-height", self.options["graph-height"]["default"])

Currently, four types of options are available:

  • OPTION_TEXT, shown as a simple text field. Optionally, min and max may be set for this option to parse the value as a (clamped) integer.
  • OPTION_TOGGLE, a checkbox. Will be either true or false.
  • OPTION_CHOICE, a select box. Select options are to be provided via options in the option definition, containing a dictionary with possible values as keys and labels for those values as values.
  • OPTION_MULTI, a list of several checkboxes. Requires an options value like OPTION_CHOICE, but the parsed value of the option will be a list of selected values rather than a single value.

Next to these type-specific option properties, the following properties may be set for all options:

  • help: The label for the option as displayed in the web interface (required)
  • default: The default value for this option (required)
  • tooltip: Further information on the option may be added through a tooltip which can be displayed through a tooltip anchor displayed next to the option in the interface (option).

🐈🐈🐈🐈

Clone this wiki locally