Building

Building the connectors from the source is only necessary when we want to use or contribute to the latest (unreleased) version of the Pravega Flink connectors.

Checkout the source code repository by following below steps:

git clone https://github.com/pravega/flink-connectors.git

After cloning the repository, the project can be built by running the below command in the project root directory flink-connectors.

./gradlew clean build -Xtest

To publish the artifacts in the local maven repository cache ~/.m2/repository, run the following command:

./gradlew clean publishToMavenLocal

Customizing the Build

Building against a custom Flink/Pravega version

Please start from an existing branch that matches the major version for both projects, e.g. r0.10-flink1.12 for Pravega 0.10.x and Flink 1.12.x. We can then check and change the Flink/Pravega version via the flinkVersion/pravegaVersion variable in the gradle.properties file.

Building against another Scala version

This section is only relevant if you use Scala in the stream processing application with Flink and Pravega.

Parts of the Apache Flink use the language or depend on libraries written in Scala. Because Scala is not strictly compatible across versions, there exist different versions of Flink compiled for different Scala versions. If we use Scala code in the same application where we use the Apache Flink or the Flink connectors, we typically have to make sure we use a version of Flink that uses the same Scala version as our application.

By default, the dependencies for the connector releases point to Scala 2.12. We need to the Scala version via the scalaVersion variable in the gradle.properties file and build a custom connector. This will help to change all entries for the Flink dependencies in the connector to have a different Scala version suffix. For example, flink-streaming-java_2.12 would be replaced by flink-streaming-java_2.11 for Scala 2.11.

In order to build a new version of Flink for a different Scala version, please refer to the Flink documentation.

Setting up your IDE

Connector project uses Project Lombok, so we should ensure that we have our IDE setup with the required plugins. (IntelliJ is recommended).

To import the source into IntelliJ:

1. Import the project directory into IntelliJ IDE. It will automatically detect the gradle project and import things correctly.
2. Enable Annotation Processing by going to Build, Execution, Deployment -> Compiler > Annotation Processors and checking Enable annotation processing.
3. Install the Lombok Plugin. This can be found in Preferences -> Plugins. Restart your IDE.
4. Connectors project compiles properly after applying the above steps.

For eclipse, we can generate eclipse project files by running ./gradlew eclipse.