Why we built ArrowLake

Early on, SAMSKARA’s storage story was the same as most notebook tools: read a CSV, write a CSV, maybe read from a database if you configured a connection. That’s fine for one person’s experiments. It falls apart the moment a second person, or a scheduled job, needs to depend on what the first person wrote — there’s no schema to check against, no history if a bad run overwrites something, no single place to see what tables actually exist.

The obvious answer is “add a lakehouse.” The less obvious part is that most lakehouse stacks assume you’re already running a Spark cluster, or are willing to start. We weren’t willing to make that the price of admission. ArrowLake is what we built instead.

The two pieces that mattered

Apache Iceberg as the table format, because we wanted real schema evolution, partitioning, and — critically — snapshot history without inventing our own. Every write through sm.write_arrowdelta becomes a real Iceberg snapshot on your own object storage. That’s what makes time travel possible:

sm.write_arrowdelta(df, "bronze.weather.readings", mode="append")

# Read the table exactly as it looked yesterday
yesterday = sm.read_arrowdelta("bronze.weather.readings", as_of="2026-07-31")

DuckDB as the query engine, because it’s the one engine we found that could query Iceberg tables directly, embed inside our own backend process, and still feel instant for exploratory SQL. No separate query service to deploy, no connection pool to a remote cluster — sm.sql_arrowdelta(...) runs in-process.

Put together: a real Iceberg catalog, queried through DuckDB, backed by whatever S3-compatible storage you already run. No Spark, no dedicated metastore cluster, nothing beyond SAMSKARA itself.

What we gave up

We’re not pretending this is free. DuckDB is not a distributed query engine — for genuinely enormous scans across huge fact tables, a real distributed engine will beat it. We made a bet that most teams in our target range (20–200 person companies) are not running queries at that scale often enough to justify the operational cost of the alternative. If that bet is wrong for your workload, ArrowLake will tell you — DuckDB’s EXPLAIN output doesn’t hide from you the way a black-box managed service can.

We also didn’t build a standalone lakehouse product. ArrowLake only exists inside SAMSKARA’s execution model — that’s a deliberate trade-off, not an oversight. It means the catalog, the notebook that wrote to it, and the SQL Workbench that reads it all share one mental model instead of three separate tools glued together after the fact.

Multi-profile, from day one

One thing we didn’t want to bolt on later: teams running more than one ArrowLake instance — one per environment, one per data domain, whatever the split is. A project can link multiple ArrowLake profiles, and SQL Workbench shows a three-level tree, catalog → namespace → table, across all of them. sm.write_arrowdelta and sm.read_arrowdelta both take a profile= argument when you need to be explicit about which one.

Where it’s headed

ArrowLake is still young. The next things on our list are lineage that reaches back into the notebook run that produced a given snapshot, and letting the AI Sparkle side of the platform reason about table history the same way it already reasons about schema. Both are extensions of the same idea: the catalog shouldn’t be a separate system you have to context-switch into.

If you’re evaluating DuckDB or Iceberg on their own and wondering what a full execution layer around them looks like, that’s exactly what ArrowLake is — take a look or book a demo and we’ll walk through it on real data.