Skip to content

Make Flink|Spark easier!!! The original intention of StreamX is to make the development of Flink easier. StreamX focuses on the management of development phases and tasks. Our ultimate goal is to build a one-stop big data solution integrating stream processing, batch processing, data warehouse and data laker.

License

Notifications You must be signed in to change notification settings

KingdomWG/streamx

Β 
Β 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 

History

73 Commits
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation


Make Flink|Spark easier!!!

English | δΈ­ζ–‡

StreamX

Make Flink|Spark easier

A magical framework that makes Flink development easier

πŸš€ Introduction

The original intention of StreamX is to make the development of Flink easier. StreamX focuses on the management of development phases and tasks. Our ultimate goal is to build a one-stop big data solution integrating stream processing, batch processing, data warehouse and data laker.

StreamX video

πŸŽ‰ Features

  • Scaffolding
  • Out-of-the-box connectors
  • Support maven compilation
  • Configuration
  • Support Applicaion and Yarn-Per-Job mode
  • start, stop, savepoint, resume from savepoint
  • Flame graph
  • Notebook
  • Project configuration and dependency version management
  • Task backup and rollback
  • Manage dependencies
  • UDF
  • Flink SQL WebIDE
  • Catalog、Hive
  • Full support from task development to deployment
  • ...

πŸ³β€πŸŒˆ Components

Streamx consists of three parts,streamx-core,streamx-pump and streamx-console

1️⃣ streamx-core

streamx-core is a framework that focuses on coding, standardizes configuration, and develops in a way that is better than configuration by convention. Also it provides a development-time RunTime Content and a series of Connector out of the box. At the same time, it extends DataStream some methods, and integrates DataStream and Flink sql api to simplify tedious operations, focus on the business itself, and improve development efficiency and development experience.

2️⃣ streamx-pump

streamx-pump is a planned data extraction component, similar to flinkx. Based on the various connector provided in streamx-core, the purpose is to create a convenient, fast, out-of-the-box real-time data extraction and migration component for big data, and it will be integrated into the streamx-console.

3️⃣ streamx-console

streamx-console is a stream processing and Low Code platform, capable of managing Flink tasks, integrating project compilation, deploy, configuration, startup, savepoint, flame graph, Flink SQL, monitoring and many other features. Simplify the daily operation and maintenance of the Flink task.

Our ultimate goal is to build a one-stop big data solution integrating stream processing, batch processing, data warehouse and data laker.

Thanks to the above excellent open source projects and many outstanding open source projects that are not mentioned, for giving the greatest respect, special thanks to Apache Zeppelin , IntelliJ IDEA, Thanks to the fire-spark project for the early inspiration and help.

πŸš€ Quick Start

git clone https://github.com/streamxhub/streamx.git
cd Streamx
mvn clean install -DskipTests -Denv=prod

click Document for more information

πŸ‘» Why not...❓

Apache Zeppelin

Apache Zeppelin is a Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Java, Scala and more.

At the same time we also need a one-stop tool that can cover development, test, package, deploy, and start. streamx-console solves these pain points very well, positioning is a one-stop stream processing platform, and has developed more exciting features (such as Flink SQL WebIDE, dependency isolation, task rollback , flame diagram etc.)

FlinkX

FlinkX is a distributed offline and real-time data synchronization framework based on flink widely used in DTStack, which realizes efficient data migration between multiple heterogeneous data sources.

StreamX focuses on the management of development phases and tasks. The streamx-pump module is also under planning, dedicated to solving data source migration, and will eventually be integrated into the streamx-console.

🍼 Feedback

You can quickly submit an issue. Before submitting, please check the problem and try to use the following contact information! Maybe your question has already been asked by others, or it has already been answered. Thank you!

You can contact us or ask questions via:

πŸ’° Donation

Are you enjoying this project ? πŸ‘‹

If you like this framework, and appreciate the work done for it to exist, you can still support the developers by donating β˜€οΈ πŸ‘Š

WeChat Pay Alipay
Buy Me A Coffee Buy Me A Coffee

πŸ† My sponsors (Coffee Suppliers)

πŸ’œ Monthly Supplier

Welcome individuals and enterprises to sponsor, your support will help us better develop the project

πŸ₯‡ Gold Supplier

πŸ₯ˆ Platinum Supplier

πŸ₯ˆ Silver Supplier

πŸ… Backers

Thank you to all our backers!

πŸ’¬ Join us

StreamX enters the high-speed development stage, we need your contribution.

Stargazers over time

Join the Group
join us

About

Make Flink|Spark easier!!! The original intention of StreamX is to make the development of Flink easier. StreamX focuses on the management of development phases and tasks. Our ultimate goal is to build a one-stop big data solution integrating stream processing, batch processing, data warehouse and data laker.

Resources

License

Security policy

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Java 35.7%
  • Scala 30.0%
  • Vue 16.1%
  • JavaScript 5.6%
  • Shell 4.0%
  • Perl 3.8%
  • Other 4.8%