Query Apache Arrow and Feather Files with SQL

Query Apache Arrow and Feather Files with SQL

Pandas and Polars often need a fast handoff format that preserves types. Apache Arrow is a language-agnostic columnar format for in-memory data, serialization, and data transport.¹ Feather V2 is the Arrow IPC file format on disk, while Feather V1 is a legacy format distinct from Arrow IPC.² DuckDB-WASM can ingest Arrow IPC streams and query Parquet files directly in the browser, so a DataFrame saved from pandas or Polars can become a SQL view without a separate conversion step.³

Arrow and Feather as inter-process data formats

Apache Arrow was designed around a columnar memory layout that supports fast sequential scans, random access, and zero-copy sharing across processes.¹ Feather V2 stores Arrow tables using Arrow IPC, so .feather and .arrow files share the same serialized columnar layout; Feather V1 is the older, non-Arrow format that DuckDB cannot read directly.²

Schema preservation across tools

When a pipeline writes either file, the schema travels with the data, which is why DESCRIBE can report column names, types, and nullability without guessing. That is especially useful after a pandas or Polars step, because the file carries the difference between INTEGER, DOUBLE, TIMESTAMP, LIST, and STRUCT columns instead of asking DuckDB to infer everything from text. For teams that move data between Python, R, and SQL, this schema fidelity means a single file can serve as the authoritative exchange format without a separate schema registry or documentation step, which reduces the risk of type mismatches when a file crosses a language boundary.

Because Arrow stores both the logical type and the physical layout, an integer column arrives as a true integer rather than a text column DuckDB must re-parse, which keeps arithmetic and joins correct on the first query. The same fidelity applies to nested LIST and STRUCT columns, so a complex record written by Polars reads back with the same structure in DuckDB without a flattening step.

How the workbench loads Arrow and Feather files

DuckDB-WASM can insert Arrow IPC stream bytes through insertArrowFromIPCStream and can query Parquet files directly after registering a file.³ DuckDB's Arrow extension consumes and produces Arrow IPC files through read_arrow; filenames ending in .arrow or .arrows can be scanned directly.⁴ Because Feather V2 is Arrow IPC, .feather follows the same path after the browser exposes the file bytes to DuckDB-WASM. The Arrow schema embedded in the file defines columns, types, and nullability, so DESCRIBE orders reports the file schema rather than inferred SQL types. This direct schema mapping means you spend less time casting columns and more time analyzing data, which is especially helpful when the file originated from a Polars or pandas pipeline that already enforced strict types.

When to reach for Arrow instead of Parquet

Arrow IPC is a good fit when the next step needs fast serialization, while Parquet is a better fit for repeated selective queries because it compresses pages and carries row-group statistics.⁵⁶ Arrow files appear most often as temporary exchange files between pipeline steps, whereas Parquet appears in data lakes and warehouse exports. If you received a .arrow or .feather file, it likely came from a tool that wanted fast read speed, such as a Polars pipeline step, a pandas to_feather() call, or DuckDB itself. The workbench handles both use cases: load the file you have, and the query experience is identical regardless of format.

Python and Polars workflows that produce Arrow files

When a pandas pipeline saves a checkpoint with df.to_feather('/tmp/checkpoint.feather'), it writes a Feather V2 Arrow IPC file that DuckDB-WASM can ingest as Arrow bytes. A Polars pipeline that writes df.write_ipc('/tmp/result.arrow') produces the same Arrow IPC format with a different extension, and DuckDB reads both identically without requiring you to rename files or convert between formats manually.

Verifying Python and Polars round trips

After loading, run DESCRIBE to confirm that column types survived the round-trip from your pipeline. If a timestamp became VARCHAR, the upstream export likely wrote text rather than an Arrow timestamp. If a nullable integer became DOUBLE, inspect the source DataFrame for nulls and consider converting to a nullable integer type before exporting. CapyToolkit keeps this inspection local in the browser, so you can validate pipeline outputs without installing DuckDB on the machine that produced the file.

Choosing between Parquet export and keeping Arrow format

For results you plan to load directly into a Python script, the Parquet file you download from the workbench is immediately readable with pd.read_parquet() or pl.read_parquet(). A round-trip from Arrow IPC in the workbench to Parquet export to Polars read is type-safe because all three steps share the Apache Arrow type system.

When to convert Arrow to Parquet for future queries

For files you will query repeatedly in future workbench sessions, converting Arrow to Parquet using the Export button produces a file that benefits from compression and row-group statistics on future selective WHERE queries.⁵⁶ Parquet embeds per-row-group min/max statistics that Arrow IPC files lack. Load the Arrow file, run SELECT * FROM tablename, export as Parquet, and use the Parquet file as your new analytical baseline. Selective queries on sorted columns can run faster because Parquet readers can discard row groups that cannot match the filter. The Export button produces a type-safe Arrow-to-Parquet round trip you can confirm locally before wiring it into a pipeline that depends on it.

When to use this

Use this when you have a .arrow or .feather file produced by pandas, Polars, R, or another Arrow-native tool and want to run SQL aggregations or filters on it without installing DuckDB locally.

Examples

DESCRIBE orders;

SELECT product_category,
       AVG(unit_price) AS avg_price,
       COUNT(*) AS n
FROM products
GROUP BY product_category
ORDER BY avg_price DESC;

SELECT *
FROM events
WHERE event_date >= '2026-03-01'
  AND user_segment = 'paid';

SELECT metadata.source,
       metadata.version,
       COUNT(*) AS n
FROM records
GROUP BY 1, 2;