BountyHub Docs

Datasets Overview

How project datasets store observations, reconcile identity, expose current state and history, and where to query, filter, view, export, and push them.

Datasets overview

A dataset is a project-owned table of observations — subdomains, live hosts, findings, or anything else your workflows collect. Observations are append-only: each push records a new snapshot with an ingest timestamp, and the service derives the current state from the history.

Open a project's Datasets tab to browse the Query, Table, Views, and Notifications surfaces. This page explains the model; the rest of the section covers each surface in depth.

#The model

Concept	Meaning
Observation	One row pushed with `step.update_dataset` or `IngestDatasetRows`.
Ingest timestamp	`__ingested_at`, recorded for every observation.
Identity	The column(s) that make two observations the same logical record.
Current state	The latest observation per identity, with stale rows removed.
History	Every observation ever pushed, including superseded ones.
View	A saved read-only SQL query over project datasets and other views.

#Current state and history

By default, queries run over the current state: for each identity, only the latest observation (by __ingested_at) is returned, and rows marked stale are excluded. A dataset without identity columns has no reconciliation key, so every pushed row is its own record.

Set includeHistory = true (UI: Include history; CLI: --include-history) to run over the raw append-only stream instead. History keeps every observation, including duplicate identities and older versions.

#Reserved columns

Every dataset relation exposes two internal columns in addition to your schema:

Column	Type	Meaning
`__ingested_at`	timestamp	When the observation was pushed.
`__deleted`	boolean	Whether the record is marked stale. Current-state queries exclude `true` rows.

Both are reserved names and cannot be added to a dataset schema.

#Removing rows

DeleteDatasetRows removes records selected by a structured filter. mode chooses what happens to the stored observations:

Mode	Effect
`DATASET_DELETE_MODE_SOFT`	Appends tombstones for the matching identities. Current-state queries hide the records; history keeps the tombstones and the observations they supersede. Requires identity columns.
`DATASET_DELETE_MODE_HARD`	Rewrites stored files to remove every observation belonging to the matching records. History no longer contains them.

ClearDataset removes every record and stored file in the table. Deleting a dataset, clearing it, or hard-deleting rows releases the parquet storage from the account's storage usage; soft deletes and new observations record the bytes they add.

#Pushing observations

Workflows push transform output into an existing dataset. The dataset service owns identity matching, reconciliation, and storage; the workflow supplies only the rows and the mode.

BHW

mode = "upsert" updates or creates records matching known identities; identities absent from the push are untouched.
mode = "sync" declares the push authoritative: identities absent from the push are marked stale.

See Pushing dataset rows for mapping, transform sources, and a complete asset-tracking workflow.

#Query surfaces

Surface	Runs over	Where
Single-dataset SQL	One dataset, current state or history	Datasets → Query → Advanced SQL
Structured filters	The result of the single-dataset query	Datasets → Query → Filters
Saved views	Project datasets and other views, current state only	Datasets → Views
Project-wide SQL	Every dataset in a project, current state only	API `QueryDatasets`
Describe	Column null counts, distinct counts, top values	CLI `bh dataset describe` or API
Export	A full single-dataset query result	Datasets query download or API

The only relations a query may reference are the ones registered for its scope: dataset for a single-dataset query, or the project's dataset names for a project-wide query. File scans, catalog tables, table functions, and other datasets are rejected. See SQL reference for the full policy.

#Dataset notifications

Each dataset can route row changes to email, Discord, or Slack channels from the dataset's Notifications tab. Create the integration and notification channel first, then assign it to the dataset and choose the routes: rows added, removed, or changed. See Notifications overview for channel setup.

#Limits

Limit	Value
Datasets per project	100
Views per project	100
Single-dataset and project-wide page size	Default 100, maximum 1,000
Saved-view page size	Default 100, maximum 10,000
Row materialization guard (engine)	100,000
Query memory	2 GB by default
Concurrent interactive queries per user	2
Interactive queries per user per minute	60
Export download URL lifetime	15 minutes

Workflow runners are exempt from the interactive fairness limits; they are bounded by runner capacity instead. Operators can tune the defaults with BOUNTYHUB_HUB_DATASET_QUERY_MAX_ROWS, BOUNTYHUB_HUB_DATASET_QUERY_MEMORY_LIMIT, BOUNTYHUB_HUB_DATASET_QUERY_MAX_CONCURRENT_PER_USER, and BOUNTYHUB_HUB_DATASET_QUERY_PER_MINUTE.

#Next steps

Schemas and identity: design a dataset before you push.
Pushing dataset rows: push from a workflow.
SQL reference: query the data with DuckDB.
Structured filters: filter without writing SQL.
Saved views: keep reusable queries in the project.
API and CLI: automate datasets outside the Hub UI.