Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Snowbridge snowplow_mode details #999

Merged
merged 3 commits into from
Sep 5, 2024
Merged

Add Snowbridge snowplow_mode details #999

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
72568b6
Add details on Snowbridge `snowplow_mode`
jethron Sep 5, 2024
744ee6d
Fix spelling/formatting on custom js transformation page
jethron Sep 5, 2024
0514e9a
Add type hints for Snowbridge custom JS sample
jethron Sep 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 47 additions & 6 deletions ...nowbridge/configuration/transformations/custom-scripts/create-a-script/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,11 @@ sidebar_position: 100

# Writing a script

Custom tranformation scripts may be defined in Javascript and provided to Snowbridge.
Custom transformation scripts may be defined in Javascript and provided to Snowbridge.

## The scripting interface

The script must define a main function with a single argument. Snowbridge will pass the engineProtocol data structure as the argument:
The script must define a main function with a single argument. Snowbridge will pass the `engineProtocol` data structure as the argument:


```go
Expand All @@ -23,9 +23,21 @@ type engineProtocol struct {

This structure is represented as an object in the script engine, and serves as both the input and output of the script.

Scripts must define a main function with a single input argument:
Scripts must define a `main` function with a single input argument (JSDoc for type information is optional):

```js
/**
* @typedef {object} EngineProtocol
* @property {boolean} FilterOut
* @property {string} PartitionKey
* @property {(string | Object.<string, ?>)} Data
* @property {Object.<string, string>} HTTPHeaders
*/

/**
* @param {EngineProtocol} input
* @return {Partial<EngineProtocol>}
*/
function main(input) {
return input
}
Expand All @@ -35,10 +47,37 @@ function main(input) {

Scripts can access the message Data at `input.Data`, and can return modified data by returning it in the `Data` field of the output. Likewise for the partition key to be used for the destination - `input.PartitionKey` and the `PartitionKey` field of the output.

By default, the Data field will be a string. For Snowplow enriched TSV data, the Javascript transformation has a `snowplow_mode` option, which transforms the data to an object first.
By default, the input's `Data` field will be a string in [enriched TSV format](/docs/understanding-your-pipeline/canonical-event/understanding-the-enriched-tsv-format/index.md).
This can be changed with the [SpEnrichedToJson](/docs/destinations/forwarding-events/snowbridge/configuration/transformations/builtin/spEnrichedToJson.md) transform, or the Javascript transformation itself has a `snowplow_mode` option, which transforms the data to an object first.

The output of the script must be an object which maps to engineProtocol.

### `snowplow_mode`

`snowplow_mode` uses the [Go Analytics SDK](/docs/destinations/analytics-sdk/analytics-sdk-go/index.md) to parse the TSV fields into an object suitable for working with in Go.
The result of the [`ParsedEvent.ToMap()`](https://pkg.go.dev/github.com/snowplow/snowplow-golang-analytics-sdk/analytics#ParsedEvent.ToMap) method is the input to your transform function.

The [keys of the resulting map](https://github.com/snowplow/snowplow-golang-analytics-sdk/blob/a3430fbe576483d615b713120cfb5e443897d572/analytics/mappings.go#L153) are defined by the Analytics SDK.
Values are primitives (string, number, boolean) or, for timestamps (e.g. `derived_tstamp`, `collector_tstamp`), [`time.Time`](https://pkg.go.dev/time#Time) objects.

:::tip

To work with native Javascript [Date](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date) objects for timestamps, construct one via `new Date(ts.UnixMilli())` where `ts` is a `time.Time` instance.

When returned as part of `Data`, `time.Time` instances will [serialize in JSON](https://pkg.go.dev/time#Time.MarshalJSON) as [RFC3339](https://www.rfc-editor.org/rfc/rfc3339.html) strings.

:::

Structured data ([Self Describing Event](/docs/understanding-your-pipeline/events/index.md#self-describing-events) payloads and [Entities](docs/understanding-your-pipeline/entities/index.md)) will have keys with a prefix of `unstruct_event_` or `contexts_`, the vendor name converted to snake case, the event/entity name in snake case, and the [schema model version](/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver/index.md). For example:

| **Schema URI** | **Type** | **Key** |
| --- | --- | --- |
| `iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0` | Entity | `contexts_com_snowplowanalytics_snowplow_web_page_1` |
| `iglu:org.w3/PerformanceNavigationTiming/jsonschema/1-0-0` | Entity | `contexts_org_w3_performance_navigation_timing_1` |
| `iglu:com.urbanairship.connect/OPEN/jsonschema/1-0-0` | Event | `unstruct_event_com_urbanairship_connect_open_1` |

Timestamps in structured data values will always have their primitive representation (string/number) and not be `time.Time` instances.

## Transforming Data

For all the below examples, the input is a string representation of the below JSON object. For Snowplow data, using `snowplow_mode` will produce a JSON object input - see [the snowplow example](../examples/snowplow/index.md).
Expand All @@ -59,7 +98,7 @@ https://github.com/snowplow/snowbridge/blob/master/assets/docs/configuration/tra

## Filtering

If the `FilterOut` field of the output is returned as `true`, the message will be acked immediately and won't be sent to the target. This will be the behavior regardelss of what is returned to the other fields in the protocol.
If the `FilterOut` field of the output is returned as `true`, the message will be acknowledged immediately and won't be sent to the target. This will be the behavior regardless of what is returned to the other fields in the protocol.

```js reference
https://github.com/snowplow/snowbridge/blob/master/assets/docs/configuration/transformations/custom-scripts/create-a-script-filter-example.js
Expand All @@ -81,12 +120,14 @@ https://github.com/snowplow/snowbridge/blob/master/assets/docs/configuration/tra

## Setting an HTTP header

For the http target only, you can specify a set of http headers, which will be appended to the configured headers for the http target. Do so by prividing an object in the `HTTPHeaders` field:
For the `http` target only, you can specify a set of HTTP headers, which will be appended to the configured headers for the `http` target. Do so by providing an object in the `HTTPHeaders` field:

```js reference
https://github.com/snowplow/snowbridge/blob/master/assets/docs/configuration/transformations/custom-scripts/create-a-script-header-example.js
```

The headers will only be included if the target has the [`dynamic_headers = true` setting](/docs/destinations/forwarding-events/snowbridge/configuration/targets/http/index.md#configuration-options) configured.

## Configuration

Once your script is ready, you can configure it in the app by following the [Javascript](/docs/destinations/forwarding-events/snowbridge/configuration/transformations/custom-scripts/javascript-configuration/index.md) configuration page.
Expand Down