Pinned Loading

1. Data-Ingestion-with-Kafka-and-NiFi Data-Ingestion-with-Kafka-and-NiFi Public

   This project demonstrates the integration of Apache Kafka, Apache NiFi, and a Python producer/consumer using confluent_kafka.

   Python

2. learning-projects learning-projects Public

   Repository for my learning porjects on data engineering and machine learning

   Python

3. parkinson-predictive-analysis parkinson-predictive-analysis Public

   This script processes the combined clinical, peptide, and protein data to train a machine learning model for predicting the severity of Parkinson's disease as measured by UPDRS scores. The script i…

   Jupyter Notebook

4. Distributed_Data_Storage Distributed_Data_Storage Public

   Distributed Data Storage with Hadoop HDFS and Amazon S3

   Shell

5. Data_Processing_using_Spark_Flink Data_Processing_using_Spark_Flink Public

   This project demonstrates data cleaning, processing with Apache Spark and Apache Flink, both locally and on AWS EMR.

   Python

6. Data_Pipeline_Spark_Azure_DBT Data_Pipeline_Spark_Azure_DBT Public

   In this project, I tried implementing a data engineering pipeline using the Medallion Architecture with a set of specific technologies: Apache Spark, Azure Databricks, Data Build Tool (DBT), and Az…