Architecture overview

A general overview of the architecture and workflow of Exynize platform

Exynize platform is based on technologies provided by modern javascript stack to make it easy to learn, develop and run custom functions and pipelines in the cloud. The core of the platform is based on node.js which provides a robust way to run user-created code in a controlled, sandboxed environment. Using npm allows users to re-use hundreds of thousands of existing packages in their code. Exynize also utilizes babel.js that is used to provide the latest possible javascript syntax and langauge features.

The core of the platform is built around RxJS that provides a great API for asynchronous programming with observable streams. RxJS allows to easily create functions that can be both synchronous or asynchronous, as well as to create source functions that are capable of dispatching more than one value over time. If one would to simplify the whole platform into one line of code, it'd look like this:

const pipeline = userProcessingFunctions
  .reduce((fn, stream) => stream.flatMap(fn), Rx.Observable.create(sourceFunction))
  .subscribe(result => renderFunction(result));

Using RxJS also allows to distribute all of the user functions that needs to be executed between several dedicated runner nodes. During execution, Exynize instantiates every user function as a small, stand-alone service. This allows to distribute intensive tasks among several VMs, balance resources in a better way, as well as to spawn multiple instances of the same processing function to enable effortless horizontal scaling.

Distribution and scaling is achieved by using Runner services which rely on RabbitMQ message bus for communication. To simplify interactions with the RabbitMQ, Exynize utilizes Microwork.js library that provides a simple abstract interface for it, as well as scaling and balancing features.

The REST API is built using Express.js and uses RethinkDB for storage. RethinkDB is also used internally in REST as message bus for communication between pipeline running child processes and the user-facing APIs.

The UI is created using React.js for rendering and RxJS for managing state. Bootstrap css framework along with Bootswatch Paper theme is used for styling. Packaging, minification and all other tasks are handled by Webpack. The same stack is also used for compilation of the user-defined render components.

WebSockets are used in cases when the real-time data transfer is required. For example, when providing live updates to rendered pipeline result.