# Datasets overview

A dataset is a project-owned table of observations — subdomains, live hosts,
findings, or anything else your workflows collect. Observations are append-only:
each push records a new snapshot with an ingest timestamp, and the service derives
the current state from the history.

Open a project's **Datasets** tab to browse the Query, Table, Views, and
Notifications surfaces. This page explains the model; the rest of the section
covers each surface in depth.

## The model

<table>
<thead>
  <tr>
    <th>
      Concept
    </th>
    
    <th>
      Meaning
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      Observation
    </td>
    
    <td>
      One row pushed with <code>
        step.update_dataset
      </code>
      
       or <code>
        IngestDatasetRows
      </code>
      
      .
    </td>
  </tr>
  
  <tr>
    <td>
      Ingest timestamp
    </td>
    
    <td>
      <code>
        __ingested_at
      </code>
      
      , recorded for every observation.
    </td>
  </tr>
  
  <tr>
    <td>
      Identity
    </td>
    
    <td>
      The column(s) that make two observations the same logical record.
    </td>
  </tr>
  
  <tr>
    <td>
      Current state
    </td>
    
    <td>
      The latest observation per identity, with stale rows removed.
    </td>
  </tr>
  
  <tr>
    <td>
      History
    </td>
    
    <td>
      Every observation ever pushed, including superseded ones.
    </td>
  </tr>
  
  <tr>
    <td>
      View
    </td>
    
    <td>
      A saved read-only SQL query over project datasets and other views.
    </td>
  </tr>
</tbody>
</table>

### Current state and history

By default, queries run over the **current state**: for each identity, only the
latest observation (by `__ingested_at`) is returned, and rows marked stale are
excluded. A dataset without identity columns has no reconciliation key, so every
pushed row is its own record.

Set `includeHistory = true` (UI: **Include history**; CLI: `--include-history`)
to run over the raw append-only stream instead. History keeps every observation,
including duplicate identities and older versions.

### Reserved columns

Every dataset relation exposes two internal columns in addition to your schema:

<table>
<thead>
  <tr>
    <th>
      Column
    </th>
    
    <th>
      Type
    </th>
    
    <th>
      Meaning
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        __ingested_at
      </code>
    </td>
    
    <td>
      timestamp
    </td>
    
    <td>
      When the observation was pushed.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        __deleted
      </code>
    </td>
    
    <td>
      boolean
    </td>
    
    <td>
      Whether the record is marked stale. Current-state queries exclude <code>
        true
      </code>
      
       rows.
    </td>
  </tr>
</tbody>
</table>

Both are reserved names and cannot be added to a dataset schema.

## Removing rows

`DeleteDatasetRows` removes records selected by a structured filter. `mode`
chooses what happens to the stored observations:

<table>
<thead>
  <tr>
    <th>
      Mode
    </th>
    
    <th>
      Effect
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        DATASET_DELETE_MODE_SOFT
      </code>
    </td>
    
    <td>
      Appends tombstones for the matching identities. Current-state queries hide the records; history keeps the tombstones and the observations they supersede. Requires identity columns.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        DATASET_DELETE_MODE_HARD
      </code>
    </td>
    
    <td>
      Rewrites stored files to remove every observation belonging to the matching records. History no longer contains them.
    </td>
  </tr>
</tbody>
</table>

`ClearDataset` removes every record and stored file in the table. Deleting a
dataset, clearing it, or hard-deleting rows releases the parquet storage from
the account's storage usage; soft deletes and new observations record the bytes
they add.

## Pushing observations

Workflows push transform output into an existing dataset. The dataset service
owns identity matching, reconciliation, and storage; the workflow supplies only
the rows and the mode.

```bhw
step.update_dataset(
    table = "subdomains",
    source = path.file("subs-clean.txt").lines(),
    pipeline = transform.trim().lower().unique(),
    mapping = {
        "host": row.value,
        "source": "subfinder",
    },
    mode = "upsert",
)
```

- `mode = "upsert"` updates or creates records matching known identities;
identities absent from the push are untouched.
- `mode = "sync"` declares the push authoritative: identities absent from the
push are marked stale.

See [Pushing dataset rows](/datasets/workflows) for mapping, transform sources,
and a complete asset-tracking workflow.

## Query surfaces

<table>
<thead>
  <tr>
    <th>
      Surface
    </th>
    
    <th>
      Runs over
    </th>
    
    <th>
      Where
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      Single-dataset SQL
    </td>
    
    <td>
      One dataset, current state or history
    </td>
    
    <td>
      Datasets → Query → <strong>
        Advanced SQL
      </strong>
    </td>
  </tr>
  
  <tr>
    <td>
      Structured filters
    </td>
    
    <td>
      The result of the single-dataset query
    </td>
    
    <td>
      Datasets → Query → <strong>
        Filters
      </strong>
    </td>
  </tr>
  
  <tr>
    <td>
      Saved views
    </td>
    
    <td>
      Project datasets and other views, current state only
    </td>
    
    <td>
      Datasets → Views
    </td>
  </tr>
  
  <tr>
    <td>
      Project-wide SQL
    </td>
    
    <td>
      Every dataset in a project, current state only
    </td>
    
    <td>
      API <code>
        QueryDatasets
      </code>
    </td>
  </tr>
  
  <tr>
    <td>
      Describe
    </td>
    
    <td>
      Column null counts, distinct counts, top values
    </td>
    
    <td>
      CLI <code>
        bh dataset describe
      </code>
      
       or API
    </td>
  </tr>
  
  <tr>
    <td>
      Export
    </td>
    
    <td>
      A full single-dataset query result
    </td>
    
    <td>
      Datasets query download or API
    </td>
  </tr>
</tbody>
</table>

The only relations a query may reference are the ones registered for its scope:
`dataset` for a single-dataset query, or the project's dataset names for a
project-wide query. File scans, catalog tables, table functions, and other
datasets are rejected. See [SQL reference](/datasets/sql) for the full policy.

## Dataset notifications

Each dataset can route row changes to email, Discord, or Slack channels from the
dataset's **Notifications** tab. Create the integration and notification channel
first, then assign it to the dataset and choose the routes: rows added, removed,
or changed. See [Notifications overview](/notifications/overview) for channel
setup.

## Limits

<table>
<thead>
  <tr>
    <th>
      Limit
    </th>
    
    <th>
      Value
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      Datasets per project
    </td>
    
    <td>
      100
    </td>
  </tr>
  
  <tr>
    <td>
      Views per project
    </td>
    
    <td>
      100
    </td>
  </tr>
  
  <tr>
    <td>
      Single-dataset and project-wide page size
    </td>
    
    <td>
      Default 100, maximum 1,000
    </td>
  </tr>
  
  <tr>
    <td>
      Saved-view page size
    </td>
    
    <td>
      Default 100, maximum 10,000
    </td>
  </tr>
  
  <tr>
    <td>
      Row materialization guard (engine)
    </td>
    
    <td>
      100,000
    </td>
  </tr>
  
  <tr>
    <td>
      Query memory
    </td>
    
    <td>
      2 GB by default
    </td>
  </tr>
  
  <tr>
    <td>
      Concurrent interactive queries per user
    </td>
    
    <td>
      2
    </td>
  </tr>
  
  <tr>
    <td>
      Interactive queries per user per minute
    </td>
    
    <td>
      60
    </td>
  </tr>
  
  <tr>
    <td>
      Export download URL lifetime
    </td>
    
    <td>
      15 minutes
    </td>
  </tr>
</tbody>
</table>

Workflow runners are exempt from the interactive fairness limits; they are
bounded by runner capacity instead. Operators can tune the defaults with
`BOUNTYHUB_HUB_DATASET_QUERY_MAX_ROWS`,
`BOUNTYHUB_HUB_DATASET_QUERY_MEMORY_LIMIT`,
`BOUNTYHUB_HUB_DATASET_QUERY_MAX_CONCURRENT_PER_USER`, and
`BOUNTYHUB_HUB_DATASET_QUERY_PER_MINUTE`.

## Next steps

- [Schemas and identity](/datasets/schema): design a dataset before you push.
- [Pushing dataset rows](/datasets/workflows): push from a workflow.
- [SQL reference](/datasets/sql): query the data with DuckDB.
- [Structured filters](/datasets/filters): filter without writing SQL.
- [Saved views](/datasets/views): keep reusable queries in the project.
- [API and CLI](/datasets/api-cli): automate datasets outside the Hub UI.