iterabledata

# Iterable Data

<!-- mcp-name: io.github.datenoio/iterabledata -->

Iterable Data is a Python library for reading and writing data files row by row in a consistent, iterator-based interface. It provides a unified API for working with various data formats (CSV, JSON, Parquet, XML, etc.) similar to `csv.DictReader` but supporting many more formats.

This library simplifies data processing and conversion between formats while preserving complex nested data structures (unlike pandas DataFrames which require flattening).

## Features

- **Unified API**: Single interface for reading/writing multiple data formats
- **Automatic Format Detection**: Detects file type and compression from filename or content (magic numbers and heuristics)
- **Format Capability Reporting**: Programmatically query format capabilities (read/write/bulk/totals/streaming/tables)
- **Support for Compression**: Works seamlessly with compressed files
- **Preserves Nested Data**: Handles complex nested structures as Python dictionaries
- **DuckDB Integration**: Optional DuckDB engine for high-performance queries with pushdown optimizations
- **Pipeline Processing**: Built-in pipeline support for data transformation
- **Encoding Detection**: Automatic encoding and delimiter detection for text files
- **Bulk Operations**: Efficient batch reading and writing
- **Native Batch Conversion**: Opt-in columnar-to-columnar transfers with projection, row-range, and batch-size selection
- **Bounded Columnar I/O**: Shared row/bulk cursors and configurable Parquet row groups keep large reads and writes bounded
- **Codec Performance Profiles**: Choose `fast`, `balanced`, or `max` compression settings with effective-setting diagnostics
- **Table Listing**: Discover available tables, sheets, and datasets in multi-table formats
- **Context Manager Support**: Use `with` statements for automatic resource cleanup
- **DataFrame Bridges**: Convert iterable data to Pandas, Polars, and Dask DataFrames with one-liner methods
- **Cloud Storage Support**: Direct access to S3, GCS, and Azure Blob Storage via URI schemes
- **Database Engine Support**: Read-only access to SQL and NoSQL databases (PostgreSQL, ClickHouse, MySQL, MongoDB, Elasticsearch, etc.) as iterable data sources
- **Atomic Writes**: Production-safe file writing with temporary files and atomic renames
- **Bulk File Conversion**: Convert multiple files at once using glob patterns or directories
- **Progress Tracking and Metrics**: Built-in progress bars, callbacks, and structured metrics objects
- **Error Handling Controls**: Configurable error policies and structured error logging; malformed input raises typed errors by default instead of reading as empty datasets
- **Security Hardening**: XXE-safe XML parsing, AST-whitelisted filter expressions, and explicit pickle trust acknowledgement
- **Performance Regression Gate**: CI-enforced baselines for representative read/convert workloads
- **Container Formats**: Stream records from TAR archives without extracting members to disk
- **Type Hints and Type Safety**: Complete type annotations with typed helper functions for dataclasses and Pydantic models
- **Lakehouse Tables**: Read and write Delta Lake, Iceberg, and DuckLake; read Hudi; experimental Apache Paimon tables plus Row/Mosaic file formats

## Supported File Types

### Core Formats

- **JSON** - Standard JSON files
- **JSONL/NDJSON** - JSON Lines format (one JSON object per line)
- **JSON-LD** - JSON for Linking Data (RDF format)
- **CSV/TSV** - Comma and tab-separated values
- **Annotated CSV** - CSV with type annotations and metadata
- **CSVW** - CSV on the Web (with metadata)
- **PSV/SSV** - Pipe and semicolon-separated values
- **LTSV** - Labeled Tab-Separated Values
- **FWF** - Fixed Width Format
- **XML** - XML files with configurable tag parsing
- **ZIP XML** - XML files within ZIP archives
- **HTML** - HTML files with table extraction

### Binary Formats

- **BSON** - Binary JSON format
- **MessagePack** - Efficient binary serialization
- **CBOR** - Concise Binary Object Representation
- **UBJSON** - Universal Binary JSON
- **SMILE** - Binary JSON variant
- **Bencode** - BitTorrent encoding format
- **Avro** - Apache Avro binary format (read & write)
- **Pickle** - Python pickle format (untrusted input is unsafe; pass `trust=True` to acknowledge)

### Columnar & Analytics Formats

- **Parquet** - Apache Parquet columnar format
- **ORC** - Optimized Row Columnar format
- **Arrow/Feather** - Apache Arrow columnar format
- **GeoParquet** - GeoParquet metadata-aware Parquet profile with geometry/CRS preservation
- **Lance** - Modern columnar format optimized for ML and vector search
- **Vortex** - Modern columnar format with fast random access
- **Paimon Row** - Apache Paimon row format for O(1) row-number access
- **Paimon Mosaic** - Apache Paimon columnar-bucket format for wide tables
- **Paimon** - Apache Paimon warehouse/catalog tables
- **Delta Lake** - Delta Lake format (read & write)
- **Iceberg** - Apache Iceberg format (read & write)
- **DuckLake** - DuckLake lakehouse tables (read & write)
- **Hudi** - Apache Hudi format (read; writes deferred)

### Database Formats

- **SQLite** - SQLite database files
- **DBF** - dBase/FoxPro database files
- **MySQL Dump** - MySQL dump files
- **PostgreSQL Copy** - PostgreSQL COPY format
- **DuckDB** - DuckDB database files

### Statistical Formats

- **SAS** - SAS data files
- **Stata** - Stata data files
- **SPSS** - SPSS data files
- **R Data** - R RDS and RData files
- **fst** - R fst columnar on-disk frames (`fst` extra; experimental)
- **PX** - PC-Axis format
- **ARFF** - Attribute-Relation File Format (Weka format)
- **LIBSVM** - Sparse labeled feature vectors (read & write)
- **NumPy** - `.npy` / `.npz` array rows (read & write; `npy` extra)

### Scientific Formats

- **NetCDF** - Network Common Data Form for scientific data
- **CDF** - NASA Common Data Format (space science)
- **HDF5** - Hierarchical Data Format
- **Zarr** - Chunked array stores (`zarr` extra; experimental)
- **XYZ** - Molecular/point coordinate tables
- **CIF** - Crystallographic Information File (`atom_site` loops; experimental)
- **PDB** - Protein Data Bank ATOM/HETATM records
- **MATLAB MAT** - MATLAB `.mat` variables (`mat` extra; experimental)
- **SEG-Y** - Seismic traces (`geophysical` extra; experimental)
- **GRIB2** - Meteorological messages (`geophysical` extra; experimental)
- **miniSEED** - Seismological waveform windows (`geophysical` extra; experimental)

### Geospatial Formats

- **GeoJSON** - Geographic JSON format
- **GeoJSON Text Sequence** - RFC 8142 line-delimited GeoJSON Features (`.geojsonl`, `.geojsons`); streaming-friendly
- **GeoPackage** - OGC GeoPackage format
- **GML** - Geography Markup Language
- **KML** - Keyhole Markup Language
- **KMZ** - KML Zipped (ZIP archive containing KML)
- **GPX** - GPS Exchange Format (waypoints, routes, tracks)
- **Shapefile** - ESRI Shapefile format
- **File Geodatabase** - ESRI FileGDB layers via Fiona (`geospatial` extra; experimental)
- **MapInfo MIF** - MapInfo Interchange Format (`geospatial` extra; experimental)
- **Esri ASCII Grid** - Raster grids as cell or row records (`.asc`)
- **ArcInfo E00** - Interchange exports (experimental subset)
- **LAS** - LiDAR point clouds (`lidar` extra; experimental)
- **BAG** - Bathymetric Attributed Grid (`hdf5` extra; experimental)
- **CZML** - Cesium CZML document packets
- **FlatGeobuf** - Streaming geospatial features with optional spatial-index filtering
- **MVT/PBF** - Mapbox Vector Tiles
- **TopoJSON** - Topology-preserving GeoJSON extension

### RDF & Semantic Formats

- **JSON-LD** - JSON for Linking Data
- **RDF/XML** - RDF in XML format
- **Turtle** - Terse RDF Triple Language
- **N-Triples** - Line-based RDF format
- **N-Quads** - N-Triples with context
- **TriG** - RDF Triple Graph format
- **N3** - Notation3 RDF format
- **TriX** - XML Triple RDF format
- **HDT** - Header-Dictionary-Triples compact RDF (`rdf` extra; experimental)

### Feed Formats

- **Atom** - Atom Syndication Format
- **RSS** - Rich Site Summary feed format

### Network Formats

- **PCAP** - Packet Capture format
- **PCAPNG** - PCAP Next Generation format

### Log & Event Formats

- **Apache Log** - Apache access/error logs
- **CEF** - Common Event Format
- **GELF** - Graylog Extended Log Format
- **WARC** - Web ARChive format
- **CDX** - Web archive index format
- **ILP** - InfluxDB Line Protocol
- **HTML** - HTML files with table extraction

### Email Formats

- **EML** - Email message format
- **MBOX** - Mailbox format
- **MHTML** - MIME HTML format

### Configuration Formats

- **INI** - INI configuration files
- **TOML** - Tom's Obvious Minimal Language
- **YAML** - YAML Ain't Markup Language
- **HOCON** - Human-Optimized Config Object Notation
- **EDN** - Extensible Data Notation

### Office Formats

- **XLS/XLSX** - Microsoft Excel files
- **XLSB** - Excel Binary format
- **ODS** - OpenDocument Spreadsheet
- **Microsoft Access** - Access `.mdb` / `.accdb` tables (`access` extra; experimental)
- **Lotus 1-2-3** - Legacy WK1 / `.123` spreadsheets (experimental)

### Business & Exchange Formats

- **EDI** - X12 / EDIFACT segment streams (experimental)
- **IATI** - Aid-transparency activity XML (`xml` extra; experimental)

### CAD Formats

- **DXF** - AutoCAD Drawing Exchange Format

### Graph Formats

- **GraphML** - Graph Markup Language
- **GEXF** - Graph Exchange XML Format
- **DOT** - GraphViz DOT format

### Sequence & Alignment Formats

- **FASTA** - Sequence format (protein/nucleotide)
- **FASTQ** - Sequence with quality format
- **SAM** - Sequence Alignment/Map (text)
- **BAM** - Binary SAM format
- **Genomic VCF/BCF** - Variant Call Format for genomic data (distinct from vCard `.vcf`; requires `bio` extra)
- **CRAM** - Reference-compressed sequence alignments (requires `alignment` extra and an explicit reference when needed)
- **BED** - BED3–BED12 genomic intervals
- **GFF3/GTF** - Ge