# Datasets API and CLI

Datasets are exposed through two ConnectRPC services:

- `bountyhub.dataset.v1.DatasetService` — datasets, ingest, SQL, describe, export.
- `bountyhub.dataset.v1.DatasetViewService` — saved views.

The generated TypeScript SDK exposes `datasetClient` and `datasetViewClient`; the
`bh` CLI wraps the most common `DatasetService` calls. In Connect JSON, `int64`
and `uint64` fields (for example `rowCount`) are encoded as strings.

## Authentication and scopes

<table>
<thead>
  <tr>
    <th>
      Access
    </th>
    
    <th>
      Scope
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      Get, list, query, project-wide query, describe, export, execute view
    </td>
    
    <td>
      <code>
        DatasetRo
      </code>
    </td>
  </tr>
  
  <tr>
    <td>
      Create, update, delete, ingest, delete rows, clear, create/update/delete view
    </td>
    
    <td>
      <code>
        DatasetRw
      </code>
    </td>
  </tr>
</tbody>
</table>

Create a Personal Access Token with the **Datasets** scope (and **Projects**
read, which it requires). See
[Personal Access Token scopes](/developer/scopes).

## DatasetService

<table>
<thead>
  <tr>
    <th>
      RPC
    </th>
    
    <th>
      Purpose
    </th>
    
    <th>
      Pagination
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        CreateDataset
      </code>
    </td>
    
    <td>
      Create a table and schema
    </td>
    
    <td>
      —
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        GetDataset
      </code>
    </td>
    
    <td>
      Read one dataset
    </td>
    
    <td>
      —
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        ListDatasets
      </code>
    </td>
    
    <td>
      List project datasets
    </td>
    
    <td>
      Default 50, max 500
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        UpdateDataset
      </code>
    </td>
    
    <td>
      Change name or description
    </td>
    
    <td>
      —
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        DeleteDataset
      </code>
    </td>
    
    <td>
      Delete the dataset and its files
    </td>
    
    <td>
      —
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        IngestDatasetRows
      </code>
    </td>
    
    <td>
      Append observations
    </td>
    
    <td>
      —
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        DeleteDatasetRows
      </code>
    </td>
    
    <td>
      Soft- or hard-delete records selected by a filter
    </td>
    
    <td>
      —
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        ClearDataset
      </code>
    </td>
    
    <td>
      Remove every record and stored file
    </td>
    
    <td>
      —
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        QueryDataset
      </code>
    </td>
    
    <td>
      One read-only SELECT over <code>
        dataset
      </code>
    </td>
    
    <td>
      Default 100, max 1,000
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        QueryDatasets
      </code>
    </td>
    
    <td>
      One SELECT over every dataset in a project
    </td>
    
    <td>
      Default 100, max 1,000
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        DescribeDataset
      </code>
    </td>
    
    <td>
      Null/distinct counts and top values
    </td>
    
    <td>
      —
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        ExportDataset
      </code>
    </td>
    
    <td>
      Stream a full query result to a file
    </td>
    
    <td>
      —
    </td>
  </tr>
</tbody>
</table>

### CreateDataset

```json
{
  "projectId": "<project-uuid>",
  "name": "subdomains",
  "description": "Resolved subdomains per program",
  "schema": {
    "columns": [
      { "name": "host", "kind": "COLUMN_KIND_TEXT", "nullable": false, "identity": true },
      { "name": "program", "kind": "COLUMN_KIND_TEXT", "nullable": false, "identity": true },
      { "name": "resolved", "kind": "COLUMN_KIND_BOOLEAN", "nullable": true }
    ]
  }
}
```

Returns `{ "id": "<dataset-uuid>" }`. Names must be unique per project, sharing
one case-insensitive namespace with saved views; up to 100 datasets are allowed.
See [Schemas and identity](/datasets/schema).

### IngestDatasetRows

```json
{
  "datasetId": "<dataset-uuid>",
  "mode": "DATASET_INGEST_MODE_UPSERT",
  "rows": [
    {
      "fields": {
        "host": { "textValue": "api.example.com" },
        "program": { "textValue": "example" },
        "resolved": { "booleanValue": true }
      }
    }
  ]
}
```

- `fields` is keyed by column name; the service maps values by schema order.
- Missing fields become `NULL`; a `NULL` in a non-nullable column fails.
- Values must match the column kind (see
[Structured filters](/datasets/filters#value-types) for the value shape).
- `mode` is `DATASET_INGEST_MODE_UPSERT` (default) or
`DATASET_INGEST_MODE_SYNC`.
- Returns `{ "rowCount": "<n>" }`.

### DeleteDatasetRows

```json
{
  "datasetId": "<dataset-uuid>",
  "mode": "DATASET_DELETE_MODE_SOFT",
  "filter": {
    "condition": {
      "column": "host",
      "op": "FILTER_OP_EQ",
      "values": [{ "textValue": "api.example.com" }]
    }
  }
}
```

`filter` is required and selects records from the current logical state.
`DATASET_DELETE_MODE_SOFT` appends tombstones and requires identity columns;
`DATASET_DELETE_MODE_HARD` removes the matching observations from stored files.
Returns `{ "deletedRowCount": "<n>", "storageDeltaBytes": "<signed bytes>" }`.

### ClearDataset

```json
{ "datasetId": "<dataset-uuid>" }
```

Removes every record and stored file. Returns `{ "deletedRowCount": "<n>", "storageDeltaBytes": "<negative bytes>" }`.

### QueryDataset

```json
{
  "datasetId": "<dataset-uuid>",
  "sql": "SELECT host, ip FROM dataset WHERE resolved",
  "pageSize": 100,
  "pageToken": "",
  "includeHistory": false,
  "ingestedAtFrom": "2026-01-01T00:00:00Z",
  "filter": {
    "condition": {
      "column": "ip",
      "op": "FILTER_OP_IS_NOT_NULL",
      "values": []
    }
  }
}
```

<table>
<thead>
  <tr>
    <th>
      Field
    </th>
    
    <th>
      Meaning
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        sql
      </code>
    </td>
    
    <td>
      One read-only SELECT; a bare <code>
        SELECT *
      </code>
      
       becomes <code>
        SELECT * FROM dataset
      </code>
      
      .
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        pageSize
      </code>
    </td>
    
    <td>
      Default 100, max 1,000.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        pageToken
      </code>
    </td>
    
    <td>
      Cursor from <code>
        nextPageToken
      </code>
      
      .
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        includeHistory
      </code>
    </td>
    
    <td>
      Query raw observations instead of current state.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        ingestedAtFrom
      </code>
      
       / <code>
        ingestedAtTo
      </code>
    </td>
    
    <td>
      Bound <code>
        __ingested_at
      </code>
      
      .
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        filter
      </code>
    </td>
    
    <td>
      Typed filter tree; see <a href="/datasets/filters">
        Structured filters
      </a>
      
      .
    </td>
  </tr>
</tbody>
</table>

The response contains `schema.columns`, `rows` (`values` in schema order), and
`nextPageToken`.

### QueryDatasets

```json
{
  "projectId": "<project-uuid>",
  "sql": "SELECT s.host, w.status_code FROM subdomains AS s JOIN \"web-services\" AS w ON s.host = w.host WHERE s.resolved",
  "pageSize": 100
}
```

Returns `{ "result": { "columns", "rows", "nextPageToken" } }`. Filter with the
SQL `WHERE` clause; the per-relation `filters` field is reserved and not
supported by the current execution path. This path has no history mode.
Project-wide SQL is API-only; the Hub UI's SQL box is scoped to one dataset.

### DescribeDataset

```json
{ "datasetId": "<dataset-uuid>" }
```

Returns one entry per schema column with:

<table>
<thead>
  <tr>
    <th>
      Field
    </th>
    
    <th>
      Meaning
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        nullCount
      </code>
    </td>
    
    <td>
      Rows where the column is <code>
        NULL
      </code>
      
      .
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        distinctCount
      </code>
    </td>
    
    <td>
      Distinct non-null values.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        topValues
      </code>
    </td>
    
    <td>
      Up to 10 most frequent non-null values, most frequent first.
    </td>
  </tr>
</tbody>
</table>

`bh dataset describe` prints the first three top values.

### ExportDataset

```json
{
  "datasetId": "<dataset-uuid>",
  "format": "DATASET_EXPORT_FORMAT_CSV",
  "sql": "SELECT * FROM dataset",
  "includeHistory": false
}
```

<table>
<thead>
  <tr>
    <th>
      Format
    </th>
    
    <th>
      Enum
    </th>
    
    <th>
      Content type
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      CSV
    </td>
    
    <td>
      <code>
        DATASET_EXPORT_FORMAT_CSV
      </code>
    </td>
    
    <td>
      <code>
        text/csv
      </code>
    </td>
  </tr>
  
  <tr>
    <td>
      JSON Lines
    </td>
    
    <td>
      <code>
        DATASET_EXPORT_FORMAT_JSONL
      </code>
    </td>
    
    <td>
      <code>
        application/x-ndjson
      </code>
    </td>
  </tr>
  
  <tr>
    <td>
      JSON array
    </td>
    
    <td>
      <code>
        DATASET_EXPORT_FORMAT_JSON
      </code>
    </td>
    
    <td>
      <code>
        application/json
      </code>
    </td>
  </tr>
</tbody>
</table>

- `sql` is required; use `SELECT * FROM dataset` for a full export.
- Filters and time bounds use the same outer semantics as `QueryDataset`.
- Exports have no page size or row cap; rows stream to the file.
- The response returns `url` (presigned, valid for **15 minutes**), `fileName`,
`contentType`, `rowCount`, and `sizeBytes`.

## DatasetViewService

<table>
<thead>
  <tr>
    <th>
      RPC
    </th>
    
    <th>
      Scope
    </th>
    
    <th>
      Notes
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        CreateDatasetView
      </code>
    </td>
    
    <td>
      <code>
        DatasetRw
      </code>
    </td>
    
    <td>
      Validates SQL, references, and cycles.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        GetDatasetView
      </code>
    </td>
    
    <td>
      <code>
        DatasetRo
      </code>
    </td>
    
    <td>
      Returns <code>
        valid
      </code>
      
       and <code>
        invalidReason
      </code>
      
      .
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        ListDatasetViews
      </code>
    </td>
    
    <td>
      <code>
        DatasetRo
      </code>
    </td>
    
    <td>
      Default 50, max 500.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        UpdateDatasetView
      </code>
    </td>
    
    <td>
      <code>
        DatasetRw
      </code>
    </td>
    
    <td>
      Changes name, description, and SQL; resets validity.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        DeleteDatasetView
      </code>
    </td>
    
    <td>
      <code>
        DatasetRw
      </code>
    </td>
    
    <td>
      Invalidates dependent views.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        ExecuteDatasetView
      </code>
    </td>
    
    <td>
      <code>
        DatasetRo
      </code>
    </td>
    
    <td>
      Default 100, max 10,000.
    </td>
  </tr>
</tbody>
</table>

`ExecuteDatasetView` returns `{ "result": { "columns", "rows", "nextPageToken" } }`. Views run over the current state and have no filter,
history, or time-range fields; see [Saved views](/datasets/views).

## bh CLI

The CLI wraps `DatasetService` for the common read and manage paths:

```bash
# List datasets in a project
bh dataset list --project-id "$PROJECT_ID" --all

# Inspect one dataset and its columns
bh dataset get --dataset-id "$DATASET_ID"
bh dataset describe --dataset-id "$DATASET_ID"

# Create with a schema file (proto JSON)
bh dataset create \
  --project-id "$PROJECT_ID" \
  subdomains \
  --description "Resolved subdomains per program" \
  --schema-json schema.json

# Rename or change the description
bh dataset update \
  --dataset-id "$DATASET_ID" \
  --name subdomains-2026 \
  --description "Subdomains discovered in 2026"

# Query current state or history
bh dataset query \
  --dataset-id "$DATASET_ID" \
  "SELECT host, ip FROM dataset WHERE resolved"

bh dataset query \
  --dataset-id "$DATASET_ID" \
  --include-history \
  "SELECT host, \"__ingested_at\" FROM dataset"

# Delete
bh dataset delete --dataset-id "$DATASET_ID"
```

`bh dataset query` prints a `next page token` line when more rows are available
and accepts `--page-size` (default 100). It has no page-token flag; use the API
(`QueryDataset`) to continue past the first page.

There are no CLI commands for project-wide SQL, saved views, exports, or row
ingestion — use the API for those.

## Errors

<table>
<thead>
  <tr>
    <th>
      Connect code
    </th>
    
    <th>
      Cause
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        UNAUTHENTICATED
      </code>
    </td>
    
    <td>
      Missing or invalid token.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        PERMISSION_DENIED
      </code>
    </td>
    
    <td>
      The PAT lacks <code>
        DatasetRo
      </code>
      
       / <code>
        DatasetRw
      </code>
      
      .
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        NOT_FOUND
      </code>
    </td>
    
    <td>
      Unknown dataset or view, or one outside the caller's projects.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        ALREADY_EXISTS
      </code>
    </td>
    
    <td>
      Dataset name already used in the project.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        INVALID_ARGUMENT
      </code>
    </td>
    
    <td>
      Invalid schema, SQL, filter, id, or page token; a view cannot be evaluated.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        RESOURCE_EXHAUSTED
      </code>
    </td>
    
    <td>
      Too many concurrent queries or too many per minute.
    </td>
  </tr>
</tbody>
</table>

## Related

- [Schemas and identity](/datasets/schema) — column kinds and identity.
- [SQL reference](/datasets/sql) — query language and limits.
- [Structured filters](/datasets/filters) — filter AST and value types.
- [Saved views](/datasets/views) — view semantics.
- [Personal Access Token scopes](/developer/scopes) — scope names.