Private Beta
.md

BountyHub Docs

Pushing and Reading Dataset Rows

Push observations into project datasets from BHW workflows with step.update_dataset, and read them back into a run with step.download_dataset.

Pushing and reading dataset rows from workflows

Bug bounty hunting is a long game. The subdomain you discover today might host the admin panel that gets you a critical in six months. Datasets let a project accumulate knowledge across runs: every scan reconciles its observations into an evolving table of assets.

#Pushing Observations: step.update_dataset

BHW

Parameters:

ParameterMeaning
tableName of an existing project dataset table
sourceWhere the data comes from — a path or typed transform expression
pipelineTyped transform producing the records to push
mapping{column: expression} projection from each pipeline record (row) into dataset columns
mode"upsert" or "sync"

The workflow only provides source, pipeline, mapping, and mode. Everything else — identity, reconciliation, stale tracking, provenance, change events — is owned by the dataset service.

Dataset names must start with a letter and contain only letters, digits, and hyphens (live-hosts, not live_hosts). Column names are identifiers and may contain letters, digits, and underscores, but not the reserved names __ingested_at or __deleted.

#Mapping

The mapping dict's keys are dataset column names; each value is an expression evaluated against the current record, bound as row. With a JSONL source, fields come straight off the record:

BHW

With line-based sources, row.value is the raw string:

BHW

#Modes

mode accepts the strings "upsert" and "sync". The dataset.upsert() / dataset.sync() values are not accepted by step.update_dataset.

#"upsert"

Create or update records matching known identities; records absent from this push are left untouched. This is what you want for accumulating observations:

BHW

#"sync"

The push represents the complete current state: identities that were known but are absent from this push become stale. Use it when the source is authoritative:

BHW

#Reading a Dataset: step.download_dataset

step.download_dataset exports the current logical state of a dataset into a file in the activity workspace, so a run can start from knowledge accumulated by previous runs:

BHW

Parameters:

ParameterMeaning
tableName of an existing project dataset table
destinationWorkspace file the export is written to (path.file(...))
format"csv" (default), "jsonl", or "json"

Later steps read the file like any other workspace input, and the step exposes it as run.output. The export covers the current logical state: latest observation per identity, stale records excluded.

#A Complete Asset-Tracking Workflow

BHW

Each night, new subdomains appear in the dataset with their source and program recorded as provenance; nothing is ever lost between runs. When a later scan uses mode = "sync" after a full validation pass, dead assets age out on their own instead of polluting your target lists forever.

#Querying the data you pushed

Pushes are only half of the feature — datasets are queried with DuckDB SQL and typed filters. The Datasets section covers schema design, SQL, filters, views, and automation: