matlab-choose-big-data-solution

# Choose Big Data Solution

Help users and agents select the right MATLAB tool for large **tabular data in
file-based formats** (CSV, Parquet, delimited text, spreadsheets, MDF). The
skill encodes a decision flowchart — recommend one clear path, not a menu of
options.

## When to Use

- User or agent has large tabular data in file-based formats (CSV, Parquet, delimited text, spreadsheets, MDF)
- User or agent says data is "large", "big", or "huge" (even without a specific size)
- User or agent hits an out-of-memory error with `readtable`, `parquetread`, or similar
- User or agent has multiple tabular files to process as a single dataset
- User or agent has multiple tabular files to process independently (per-file)
- User or agent asks how to scale up an existing workflow on tabular file data
- User or agent asks about datastores, tall arrays, or mapreduce
- User or agent needs to build a custom datastore class for a proprietary or non-standard format
- User or agent has large XML, JSON, HTML, or Word document files — `readtable` supports these formats but the built-in datastores do not. Scaling these requires a custom datastore (see "Formats Without a Built-in Datastore" and "Custom Datastores" sections below)

## When NOT to Use

- Single file that fits in memory (file-size-to-RAM ratio < 0.5) — see Pre-Flight Check and `matlab-import-export-data` skill
- User or agent is working with MAT files — `matfile` provides partial I/O for large `.mat` files, different workflow
- User or agent is working with databases (SQL, ODBC) — use Database Toolbox. See the `matlab-use-database` and `matlab-use-duckdb` skills in the `reporting-and-database-access` category
- User or agent needs GPU acceleration — different domain
- Deep learning training workflows — datastore is correct for data loading, but use `minibatchqueue` for batching (not tall or transform)

## Pre-Flight Check

**ALWAYS run this check BEFORE recommending datastore or tall array patterns.**
Estimate the file-size-to-available-RAM ratio (accounting for in-memory
expansion of the file format).

| Condition | Route | Rationale |
|-----------|-------|-----------|
| Single file, ratio < 0.5 | Native MATLAB I/O (`readtable`, `parquetread`) | Fits in memory; datastore/tall adds unnecessary complexity |
| Single file, ratio ≥ 0.5, OR user/agent reports OOM | Continue to Decision Flowchart below | Data may not fit in memory |
| Multiple files | Continue to Decision Flowchart below | Datastore patterns provide unified multi-file access |
| Size unknown and user/agent describes data as "large", "big", or "huge" | Continue to Decision Flowchart below | Assume large until proven otherwise |

**If the data fits in memory (single file, ratio < 0.5):** recommend native
I/O and stop. Mention that datastore and tall array patterns exist if the
data grows beyond memory in the future, but do not implement them now.

## Decision Flowchart

Follow this flowchart strictly. Present ONE recommended path, not multiple
alternatives. Only mention alternatives if the situation is ambiguous.

```
Is the data described as "large" or causing OOM?
│
├── YES
│   │
│   ├── Is the goal to process all data as ONE continuous dataset?
│   │   │
│   │   └── YES → Datastore + Tall Arrays
│   │             Choose datastore by format:
│   │               CSV/delimited text → tabularTextDatastore
│   │               Parquet            → parquetDatastore
│   │               Excel (.xlsx/.xls) → spreadsheetDatastore
│   │               MDF (.mf4/.mdf)    → mdfDatastore (requires Vehicle Network Toolbox)
│   │               Other formats      → Custom Datastore
│   │
│   └── Is the goal to process each unit INDEPENDENTLY?
│       │
│       ├── One read = one FILE
│       │     CSV/delimited text → tabularTextDatastore
│       │     Excel (.xlsx/.xls) → spreadsheetDatastore
│       │     Other formats      → Custom Datastore or fileDatastore
│       │
│       └── One read = one ROW GROUP
│             Parquet → parquetDatastore
│
└── NO / UNCLEAR
    └── STOP. Ask: (1) what is the file format? (2) do you need to process
        all data as one dataset, or each file/unit independently?
        Do NOT show code until these are answered.

Optional (requires Parallel Computing Toolbox):
├── Speed up tall arrays locally → Open a parallel pool
├── Speed up readall on transforms → UseParallel
└── Speed up tall arrays on Hadoop/Spark → mapreducer (also requires MATLAB Parallel Server)
```

**Critical rules:**
- **When the processing goal is unclear (continuous vs per-file), STOP and ask
  before writing any code.** Do not show code examples, do not generate scripts,
  do not demonstrate both approaches. Ask which applies and wait for the answer.
  The correct response to ambiguity is a short clarifying question, not a menu
  of options with code for each.
- When data is described as "large", NEVER lead with `readtable` or `parquetread`
- For per-file or per-row-group processing, NEVER suggest tall arrays (tall merges all data into one dataset and has no concept of file or row group boundaries)
- For Parquet, the natural independent unit is the **row group**, not the file — `parquetDatastore` defaults to `ReadSize = "rowgroup"`. Do NOT force `ReadSize = "file"` on a Parquet datastore unless the workflow truly needs whole-file granularity
- For parallelism, recommend parallel pool FIRST; mapreducer is only for Hadoop/Spark
- Do NOT present manual chunking (while/read loops) as the first option — tall arrays handle chunking automatically
- Do NOT present multiple options — follow the flowchart and recommend ONE clear path

## Datastore + Tall Arrays (continuous dataset)

Use when: processing one large file OR multiple files as a single dataset.
Use the datastore matching the format (see Decision Flowchart above).

```matlab
% Single file
ds = tabularTextDatastore("largedata.csv");
tt = tall(ds);

% Multiple files
ds = tabularTextDatastore("data/*.csv");
tt = tall(ds);

% For example, compu