BountyHub Docs

Dataset Schemas and Identity

Dataset column kinds, naming rules, nullable columns, identity columns and reconciliation, reserved fields, and how to create a dataset schema.

Datasets: schemas and identity

A dataset schema defines the columns, types, nullability, and identity of a project table. The schema is fixed when the dataset is created; rows are validated against it on every push.

#Column kinds

Kind	Stored as	Accepted values
`COLUMN_KIND_TEXT`	`TEXT`	Any UTF-8 string.
`COLUMN_KIND_INTEGER`	`BIGINT`	64-bit signed integer.
`COLUMN_KIND_NUMBER`	`DOUBLE`	Double-precision number.
`COLUMN_KIND_BOOLEAN`	`BOOLEAN`	`true` or `false`.
`COLUMN_KIND_TIMESTAMP`	`TIMESTAMP`	Timestamp with microsecond precision.

In SQL, these appear as TEXT, BIGINT, DOUBLE, BOOLEAN, and TIMESTAMP. See SQL reference for how query result types are mapped.

#Schema rules

Rule	Detail
Column names	ASCII letters, digits, and underscores, starting with a letter or underscore. Quote names that are not lowercase in SQL.
Uniqueness	Column names are unique case-insensitively.
Reserved names	`__ingested_at` and `__deleted` are reserved.
Column order	Significant: ingest and result mapping use schema order.
Nullable	A `NULL` written to a non-nullable column fails the push.
Column types	Fixed after creation; the Hub dialog states this explicitly.
Dataset names	Start with a letter; use letters, digits, and dashes (for example `live-hosts`). Dataset tables and saved views share one namespace per project, so names must be unique case-insensitively across both.
Name length	Up to 255 characters.
Description	Up to 1024 characters.

Dataset names with dashes must be double-quoted in SQL: SELECT * FROM "live-hosts".

Identity columns tell the dataset which observations describe the same logical record. With identity configured, the current state keeps only the latest observation per identity (by __ingested_at). Without identity columns, every pushed row is a separate record.

Use the smallest stable key:

Dataset	Identity
Subdomains	`host` (optionally `program` if the same host appears in several programs)
Live hosts	`url`
Findings	`fingerprint`
DNS history	usually no identity — every observation is meaningful

Identity is part of the schema, not the push:

json

#Creating a dataset

The Hub's New dataset dialog covers the common case and creates datasets without identity columns, which is correct for append-only history and for tables where every row is distinct.

When you need reconciliation, create the dataset with the API or CLI and mark the identity columns:

bash

schema.json uses the proto JSON shape:

json

The same shape is the schema field of CreateDatasetRequest — see API and CLI.

#Current state and stale rows

The service keeps every push as append-only observations and derives the current state per identity. The relation also exposes two reserved columns:

Column	Meaning
`__ingested_at`	When the observation was recorded.
`__deleted`	Whether the record is stale. Current-state queries exclude stale rows.

upsert leaves identities that are absent from a push untouched. sync declares the push authoritative, so identities missing from a full snapshot are marked stale. DeleteDatasetRows accepts the same current-state semantics: soft deletion writes __deleted = true tombstones and hard deletion rewrites the stored files. See Overview.

In SQL, you can inspect the history directly:

sql

#Changing a schema

Column names and types cannot be changed after creation, and the update API only changes the dataset name and description. To change the shape:

Create a new dataset with the desired schema.
Backfill or re-push the rows.
Point workflows and views at the new dataset name.
Delete the old dataset when you no longer need its history.

Deleting a dataset removes its stored files. Views that referenced it fail on their next execution and are marked invalid.

#Limits

Limit	Value
Datasets per project	100
Columns per dataset	No fixed server limit; keep schemas narrow for query speed
Column name length	255 characters in the Hub UI
Description length	1024 characters

#Next steps

Pushing dataset rows: push rows with step.update_dataset.
SQL reference: query the schema you just designed.
Structured filters: type-aware filtering without SQL.
Saved views: reusable queries across datasets.
API and CLI: create and inspect datasets programmatically.