The data warehouse is not dead. But it is unrecognizable.

What used to be a monolithic proprietary system — an island of transformed data, accessible only through a specific query engine, locked to one vendor — has been decomposed. The storage layer is now separate from compute. The table format is open. The transformation logic is version-controlled. The catalog is federated.

In 2026, the Modern Data Stack has matured into something more principled and more complex than the buzzword-heavy version we discussed in 2022. Let me break down what it actually looks like.

Photo by Luke Chesser on Unsplash

The Lakehouse Architecture: What It Means in Practice

The “lakehouse” concept — combining the scale and openness of a data lake with the reliability and query performance of a data warehouse — has become the de facto architecture for companies with serious data workloads.

The enabling technology: open table formats, most importantly Apache Iceberg.

Why Apache Iceberg Won

Three formats competed for the open table format standard: Delta Lake (Databricks), Apache Hudi (Uber), and Apache Iceberg (Netflix/Apple). By mid-2025, Iceberg had achieved decisive ecosystem dominance.

What Iceberg provides:

Raw Storage (S3/GCS/ADLS)
    └── Parquet/ORC data files
    └── Iceberg metadata layer
         ├── Catalog (table names → metadata pointers)
         ├── Metadata files (schema, partition info, snapshots)
         └── Manifest files (file lists with statistics)

ACID transactions on object storage: Multiple writers can commit without corrupting data. Time travel is built in. Schema evolution is safe.

Query engine independence: The same Iceberg table can be queried by Spark, Trino, DuckDB, Snowflake, BigQuery, StarRocks, or any other Iceberg-compatible engine. You’re not locked to any single query engine.

Partition evolution: Change how data is partitioned without rewriting all historical data.

-- Create an Iceberg table
CREATE TABLE events (
    id BIGINT,
    user_id BIGINT,
    event_type STRING,
    properties MAP<STRING, STRING>,
    created_at TIMESTAMP
)
USING ICEBERG
PARTITIONED BY (days(created_at));

-- Time travel: query data as of 7 days ago
SELECT * FROM events
FOR SYSTEM_TIME AS OF TIMESTAMP '2026-02-28 00:00:00';

-- Evolve the partition strategy without rewriting data
ALTER TABLE events
REPLACE PARTITION FIELD days(created_at) WITH hours(created_at);

The result: you can store 10 years of raw event data in S3 at $23/TB/month and query it from any engine without vendor lock-in.

dbt in 2026: From Transformation Tool to Data Platform

dbt (data build tool) started as a way to write SQL transformations that get version-controlled. It has evolved into something closer to a full data development platform.

What dbt Does

Raw source data (Iceberg/warehouse)
    └── dbt models (SELECT statements in .sql files)
         ├── Staging models (rename, cast, clean)
         ├── Intermediate models (joins, business logic)
         └── Mart models (final analytics tables)
    └── Tests, documentation, lineage

Each model is a SELECT statement. dbt compiles it to a CREATE TABLE or CREATE VIEW statement and handles the dependency graph, incremental loading, and test execution.

-- models/staging/stg_events.sql
SELECT
    id,
    user_id,
    event_type,
    properties:page_url::STRING AS page_url,
    created_at::TIMESTAMP AS occurred_at
FROM 
WHERE created_at >= '2020-01-01'  -- Skip ancient garbage data

-- models/marts/user_activity_daily.sql
WITH daily_events AS (
    SELECT
        user_id,
        DATE_TRUNC('day', occurred_at) AS activity_date,
        COUNT(*) AS event_count,
        COUNT(DISTINCT event_type) AS unique_event_types
    FROM 
    GROUP BY 1, 2
)
SELECT
    u.user_id,
    u.email,
    d.activity_date,
    d.event_count,
    d.unique_event_types
FROM daily_events d
JOIN  u USING (user_id)

dbt’s New Capabilities

dbt Mesh: Teams can define cross-project dependencies, allowing large organizations to break their monolithic dbt project into domain-owned sub-projects while maintaining shared models.

# In a consuming project — reference a model from another project
models:
  - name: user_lifetime_value
    columns:
      - name: user_id
        data_tests:
          - not_null
          - relationships:
              to: ref('platform', 'dim_users')  # Cross-project reference
              field: id

dbt + AI (dbt Copilot): Write a description of what you want, get a dbt model. It’s genuinely useful for standard patterns — aggregations, joins, slowly-changing dimensions. Less useful for complex business logic.

Semantic Layer: Define metrics once in dbt, query them from any BI tool. The metric definition lives in code, not scattered across five different dashboard tools.

# models/metrics/revenue_metrics.yml
metrics:
  - name: monthly_recurring_revenue
    label: MRR
    description: "Monthly recurring revenue from active subscriptions"
    model: ref('fct_subscriptions')
    calculation_method: sum
    expression: mrr_amount
    timestamp: subscription_date
    time_grains: [month, quarter, year]
    dimensions:
      - plan_type
      - country
      - acquisition_channel

The Streaming Lakehouse: Near-Real-Time Analytics

The traditional batch-processed data warehouse had a fundamental limitation: data was always hours or days old. The streaming lakehouse closes that gap.

Photo by Alexandre Debiève on Unsplash

The architecture:

Events → Kafka/Kinesis
    → Flink/Spark Streaming (transforms)
    → Iceberg tables (with streaming writes)
    → Query engines (seconds of latency)

Apache Flink has become the standard streaming processor for this pattern. Its Iceberg sink allows streaming writes while maintaining ACID semantics:

# Flink → Iceberg streaming write
from pyflink.table import TableEnvironment

env = TableEnvironment.create(...)

# Read from Kafka
env.execute_sql("""
    CREATE TABLE kafka_events (
        user_id BIGINT,
        event_type STRING,
        event_time TIMESTAMP(3),
        WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND
    ) WITH (
        'connector' = 'kafka',
        'topic' = 'user-events',
        'format' = 'json'
    )
""")

# Write to Iceberg with streaming semantics
env.execute_sql("""
    CREATE TABLE iceberg_events (
        user_id BIGINT,
        event_type STRING,
        event_time TIMESTAMP(3)
    ) WITH (
        'connector' = 'iceberg',
        'catalog-name' = 'prod_catalog',
        'catalog-database' = 'events',
        'catalog-table' = 'user_events'
    )
""")

# Stream from Kafka to Iceberg
env.execute_sql("INSERT INTO iceberg_events SELECT * FROM kafka_events")

The result: data lands in queryable Iceberg tables within 30-60 seconds of the originating event. For analytics use cases, that’s effectively real-time.

The Query Engine Landscape

With open table formats, you can choose your query engine independently of your storage. The current landscape:

DuckDB deserves special attention. It has become the standard for local data work — you can query Iceberg tables on S3 directly from your laptop with zero infrastructure:

import duckdb

# Query Iceberg on S3 from your laptop — no servers needed
conn = duckdb.connect()
conn.execute("INSTALL iceberg; LOAD iceberg;")

result = conn.execute("""
    SELECT
        event_type,
        COUNT(*) as event_count,
        APPROX_COUNT_DISTINCT(user_id) as unique_users
    FROM iceberg_scan('s3://my-data-lake/events/user_events/')
    WHERE created_at >= CURRENT_DATE - INTERVAL 7 DAYS
    GROUP BY event_type
    ORDER BY event_count DESC
""").fetchdf()

AI in the Data Stack: What’s Real

The most overhyped and most misunderstood component of the modern data stack is AI. Let me separate signal from noise.

Real and useful:

• Text-to-SQL: Natural language queries that get translated to SQL. Works well for standard analytical queries. Requires good catalog documentation (dbt docs help enormously).
• Anomaly detection: ML models that flag unusual patterns in metrics. Genuinely catches issues humans would miss.
• Data quality: AI-assisted schema inference, duplicate detection, PII identification in new datasets.
• Documentation generation: AI-written column descriptions and table summaries, seeded from sample data and column names.

Not ready for production:

• Fully autonomous data pipelines that “figure out the schema and transformations by themselves”
• AI that replaces data engineers for complex transformation logic
• Natural language as the primary interface for business analysts (most still prefer dashboards)

The practical recommendation: use AI to accelerate data engineers, not to replace them. Text-to-SQL for exploration, AI-generated docs to fill the documentation gap, anomaly detection to catch data quality issues early.

The Stack We Recommend in 2026

For a company processing 100GB-10TB/day:

Total infrastructure cost for this stack at medium scale: roughly $3,000-8,000/month on AWS, compared to $25,000-50,000/month for equivalent Snowflake + Fivetran + Looker.

The Bottom Line

The Modern Data Stack in 2026 is both more powerful and more complex than it was three years ago. Open table formats have delivered on their promise of vendor independence. dbt has matured from a transformation tool into a data development platform. The streaming lakehouse has closed the gap between real-time and batch analytics.

The teams that are winning have three things in common:

1. They invested in data quality and testing from day one (dbt tests, Great Expectations)
2. They treat data as a product with SLAs and ownership, not a byproduct of engineering work
3. They resist adding new tools until they’ve exhausted what the existing tools can do

The data stack is rich enough now that almost any analytical problem can be solved with open-source tools. The limiting factor is no longer technology — it’s discipline, data culture, and the unglamorous work of keeping pipelines healthy and data trustworthy.

That part, no amount of tooling will automate away.

이 글이 도움이 되셨다면 공감 및 광고 클릭을 부탁드립니다 :)