refresh-semantic-model

# Refreshing Semantic Models

Trigger, monitor, validate, and troubleshoot semantic model refreshes via the Power BI Enhanced Refresh REST API and Fabric CLI.

## Core Concepts

A semantic model refresh reloads data from upstream sources and/or recalculates dependent objects (calculated columns, calculated tables, measures). The scope can be the entire model, specific tables, or individual partitions.

Six refresh types are available via the REST API; a seventh (`add`) is TMSL-only:

| Type          | Reloads Data | Recalculates | Primary Use Case                     | API  |
|---------------|:------------:|:------------:|--------------------------------------|:----:|
| `full`        | Yes          | Yes          | Complete reload from scratch         | REST |
| `automatic`   | Conditional  | Conditional  | Smart refresh; process only if needed| REST |
| `dataOnly`    | Yes          | No*          | Reload data; clear dependents        | REST |
| `calculate`   | No           | Yes          | Recalculate without reloading data   | REST |
| `clearValues` | No           | No           | Empty data from objects              | REST |
| `defragment`  | No           | No           | Clean up column dictionaries         | REST |
| `add`         | Append       | Yes          | Append rows to a partition           | TMSL |

*`dataOnly` clears dependent objects (calculated columns, calculated tables) but does not recalculate them. Follow with a `calculate` refresh to restore them.

For detailed descriptions, behavior with incremental refresh policies, commit modes, and parallelism options, consult **`references/refresh-types.md`**.

## Refresh Workflow

### Step 1: Resolve IDs

Extract the workspace and model GUIDs needed for API calls:

```bash
WS_ID=$(fab get "WorkspaceName.Workspace" -q "id" | tr -d '"')
MODEL_ID=$(fab get "WorkspaceName.Workspace/ModelName.SemanticModel" -q "id" | tr -d '"')
```

### Step 2: Query Baseline Data (Pre-Refresh Validation)

Before triggering the refresh, capture a baseline snapshot to later verify that data actually changed. Execute a DAX query against the model to get current row counts or max dates:

```bash
fab api -A powerbi "groups/$WS_ID/datasets/$MODEL_ID/executeQueries" \
  -X post -i '{
    "queries": [{"query": "EVALUATE ROW(\"RowCount\", COUNTROWS(FactSales), \"MaxDate\", MAX(FactSales[OrderDate]))"}],
    "serializerSettings": {"includeNulls": true}
  }'
```

Record the output. This baseline is compared after refresh to confirm new data arrived.

### Step 3: Trigger the Refresh

**Full model refresh (simplest):**

```bash
fab api -A powerbi "groups/$WS_ID/datasets/$MODEL_ID/refreshes" \
  -X post -i '{"type":"full"}'
```

**Refresh specific tables:**

```bash
fab api -A powerbi "groups/$WS_ID/datasets/$MODEL_ID/refreshes" \
  -X post -i '{
    "type": "full",
    "objects": [{"table": "FactSales"}, {"table": "DimProduct"}]
  }'
```

**Refresh specific partitions:**

```bash
fab api -A powerbi "groups/$WS_ID/datasets/$MODEL_ID/refreshes" \
  -X post -i '{
    "type": "full",
    "objects": [
      {"table": "FactSales", "partition": "FactSales_2024"},
      {"table": "FactSales", "partition": "FactSales_2023"}
    ]
  }'
```

**Data-only refresh (skip recalculation):**

```bash
fab api -A powerbi "groups/$WS_ID/datasets/$MODEL_ID/refreshes" \
  -X post -i '{"type":"dataOnly","objects":[{"table":"FactSales"}]}'
```

**Calculate only (no data reload):**

```bash
fab api -A powerbi "groups/$WS_ID/datasets/$MODEL_ID/refreshes" \
  -X post -i '{"type":"calculate"}'
```

**Clear values from a table:**

```bash
fab api -A powerbi "groups/$WS_ID/datasets/$MODEL_ID/refreshes" \
  -X post -i '{"type":"clearValues","objects":[{"table":"StagingTable"}]}'
```

For the script-based approach with CLI arguments, use **`scripts/refresh_model.py`**.

### Step 4: Monitor Status

```bash
fab api -A powerbi "groups/$WS_ID/datasets/$MODEL_ID/refreshes?\$top=1"
```

Status values: `Unknown`, `InProgress`, `Completed`, `Failed`, `Disabled`, `Cancelled`

### Step 5: Post-Refresh Validation

After the refresh completes, re-run the same DAX query from Step 2 and compare:

```bash
fab api -A powerbi "groups/$WS_ID/datasets/$MODEL_ID/executeQueries" \
  -X post -i '{
    "queries": [{"query": "EVALUATE ROW(\"RowCount\", COUNTROWS(FactSales), \"MaxDate\", MAX(FactSales[OrderDate]))"}],
    "serializerSettings": {"includeNulls": true}
  }'
```

**If data has not changed** after a successful refresh:
- The upstream data source has not been updated
- The ETL pipeline (Fabric pipeline, notebook, Data Factory, or other orchestration) needs to run first
- Check the lakehouse/warehouse/SQL database to verify fresh data exists
- For Fabric lakehouses: run `fab run "Workspace.Workspace/Pipeline.DataPipeline"` or trigger the notebook
- The refresh only pulls what the source provides; if the source is stale, the refresh will succeed but show no new data

### Step 6: Cancel (if needed)

To cancel an in-progress enhanced refresh, first retrieve the `requestId` from the refresh history:

```bash
fab api -A powerbi "groups/$WS_ID/datasets/$MODEL_ID/refreshes?\$top=1"
```

The response includes a `requestId` field. Use it to cancel:

```bash
fab api -A powerbi "groups/$WS_ID/datasets/$MODEL_ID/refreshes/<requestId>" \
  -X delete
```

Only works for refreshes triggered via the Enhanced API (not scheduled or portal refreshes).

## Using the Refresh Script

The **`scripts/refresh_model.py`** script wraps the Enhanced Refresh API with CLI arguments:

```bash
# Full refresh
uv run scripts/refresh_model.py -w $WS_ID -m $MODEL_ID

# Refresh specific tables
uv run scripts/refresh_model.py -w $WS_ID -m $MODEL_ID --tables Sales,Calendar

# Refresh specific partitions
uv run scripts/refresh_model.py -w $WS_ID -m $MODEL_ID --partitions Sales:Sales_2024

# Data-only then calculate (two-phase)
uv run scripts/refresh_model.py -w $WS_ID -m $MODEL_ID -t dataOnly --tables FactSales
uv run scripts/refresh_model.py -