Module 1 · Conceptual Foundations
Lakehouse Architecture Foundations
Traditional architectures replicate enterprise telemetry across several stages (Raw landing ➔ Stage ➔ Core warehouse ➔ Data marts). This results in massive cloud compute charges (the Spark Tax) and security vulnerabilities due to scattered copies.
meldra.py Zero-Copy Strategy: Ingest raw CSV, API feeds, or database logs directly to open Apache Iceberg tables stored on your own S3/GCS. Queries run serverless in DuckDB or PyIceberg, keeping the storage completely decentralized and zero-copy.
# Concept: metadata-only reads
import pyiceberg.catalog
catalog = pyiceberg.catalog.load_catalog("local")
table = catalog.load_table("default.smartmeter_readings")
Module 2 · Connection Configuration
Setting up AWS Glue & GCP BigLake Catalogs
In this module, you connect meldra.py directly to your real storage catalogs. Open the product console and navigate to the Workspace tab:
- AWS Glue: Provide the S3 warehouse URI (e.g.
s3://my-company-lakehouse/), Region, and AWS Access Key credentials.
- GCP BigLake: Provide the Google Cloud Storage bucket path and upload your Service Account Key JSON.
Applying parameters establishes connection tokens. Any table created hereafter becomes queryable by external engines like Athena or BigQuery.
Module 3 · Ingestion Pipelines
Ingesting Data Tables (Append vs. Merge)
To populate your lakehouse, select the Ingest tab in the main application:
- Upload a CSV file (e.g., sample transaction ledger or customer CSV).
- Select write mode:
Append: Appends new rows to the Iceberg table on every run.
Merge: Upserts records on a unique primary key, preventing duplicate entries on S3 automatically.
# Under-the-hood: pyiceberg transaction commit
with table.transaction() as tx:
tx.append_row_groups(df.to_coo())
Module 4 · Schema Evolution
Zero-Copy Schema Evolution
One of Iceberg's greatest strengths is schema evolution without rewrite. In the Data Studio tab:
- Select a table, click the **Schema Evolution** subtab.
- Choose "Add Column" or "Rename Column" or "Drop Column".
- Meldra executes this in milliseconds as a metadata transaction, leaving your underlying parquet data files completely untouched.
Module 5 · Production automation
Data Contracts & Airflow Automation
Ensure data reliability by establishing **Data Quality Contracts** under the Data Studio subtab:
- Add rules such as
amount >= 0 or status IS NOT NULL. Meldra's ingestion engine will reject faulty records during future batch ingestion runs.
- Go to the **Workspace** tab and download the **Airflow DAG python code**. This lets you schedule daily Iceberg metadata compactions in production with zero effort.