talend-project-big-data

Data Warehouse Alimentation Project This project implements a complete data warehouse solution using Talend Open Studio for Data Integration and PostgreSQL. The system processes CSV files containing sales data (clients, products, categories, etc.) through an ETL pipeline that loads data into an Operational Data Store (ODS) and then transforms it into a dimensional data warehouse. Project Structure Architecture The project follows a typical data warehouse architecture with three main layers: Source Layer CSV files containing business data with pattern: ICOMYYYYMMDD[ENTITY].csv Located in csvFiles/ directory Entities include: categories, customers, products, sub-categories, customer types, and sales data ODS Layer (Operational Data Store) Staging area for raw data processing using tFileList components Data validation and cleansing Duplicate detection and removal Comprehensive logging with routines.system.ResumeUtil DWH Layer (Data Warehouse) Dimensional model with facts and dimensions Optimized for analytical queries Historical data preservation Database Setup PostgreSQL Configuration Create Database: Database Connections: VENTEODS: Connection for ODS schema