BountyHub Docs

Transforms

Typed data pipelines for filtering, normalizing, and partitioning scan output with step.transform.

Recon and scanning produce messy text: duplicate subdomains, mixed case, blank lines, noise you did not ask for. BountyHub workflows treat data transformation as a first-class, typed pipeline primitive instead of a pile of awk | sort | uniq shell glue.

#The Shape of a Transform

A transform is a source plus an ordered chain of operations:

BHW

source — how to read the input: path.file("subs.txt").lines() parses the file as newline-delimited lines. Other sources: .text(), .json(), .jsonl().
pipeline — the operations applied to each element (or the collection).
destination — where the result is written; must be path.file(...).

The equivalent explicit form passes the source separately:

BHW

Both forms compile to the same expression. Use whichever reads better to you.

#Sources

Source	Parses input as	Produces
`path.file(f).lines()` / `transform.lines()`	Newline-delimited lines	`Items<string>`
`path.file(f).text()` / `transform.text()`	Whole file as one string	`string`
`path.file(f).json()` / `transform.json()`	One JSON document	`json`
`path.file(f).jsonl()` / `transform.jsonl()`	Newline-delimited JSON records	`Items<record>`

record values support field access via .field, which is how JSONL output from tools like httpx -jsonl becomes structured data. Generic collection ops like .unique() work on any item type:

BHW

A bare path.file("x") without a parse is just a file reference — transforms reject it ("raw file"). Always choose a source first.

#Operations

Operations are chained fluently and are statically type-checked.

#Element-wise string ops

Applied to every item:

Op	Effect
`.trim()`	Strip surrounding whitespace
`.lower()` / `.upper()`	Change case
`.replace(from, to)`	Replace substring
`.regex(pattern)`	Extract regex match (optionally a capture group)
`.regex_replace(pattern, replacement)`	Substitute regex matches

BHW

#Collection ops

Applied to the whole stream:

Op	Effect
`.unique()`	Remove duplicates
`.take(n)` / `.drop(n)`	Head/tail slicing
`.sort()` / `.reverse()`	Order the stream
`.split(sep)`	Split each item into multiple items
`.join(sep)`	Join all items into one string

BHW

#Lambda ops: `map` / `filter` / `flat_map`

The only way to compute per-item values or drop records by predicate. Each takes an ordinary one-parameter function (lambda or named def), captured deferred and applied at runtime:

BHW

Named functions work too (transform.map(add_port) where def add_port(host): ...). Signatures are checked: map takes exactly one parameter.

#Execution model

Ops fall into four execution kinds; knowing them predicts memory behavior:

Kind	Ops	Behavior
Element-wise (streaming)	`trim`, `lower`, `upper`, `replace`, `regex`, `regex_replace`, `map`	One item in, zero/one out; constant memory
Streaming selection/expansion	`filter`, `flat_map`, `split`	Zero or more out per item; constant memory
Stateful (streaming)	`take(n)`, `drop(n)`, `unique()`	Small buffered state; `unique` memory grows with the unique set
Barrier (buffering)	`sort`, `reverse`, `join(sep)`	Buffers the whole sequence before emitting

Prefer streaming ops for 50k-host lists; reserve sort/reverse/join for bounded outputs.

#Types Are Checked Statically

Every value in a transform has a semantic type:

String, Number, Bool, File, Directory, Json, Record, Items<T>, Batch<T>

Each operation declares which types it accepts, so mistakes fail at save time, not at 3 AM mid-scan:

BHW

Collections serialize newline-delimited on write, so Items<string> writes one value per line.

#Partitioning

Large datasets can be split into partitions for bounded parallel execution. Partition strategies come from the partition namespace:

partition.count(n) — approximately n partitions; sizes differ by at most one.
partition.items(n) — at most n items per partition.

BHW

Rules:

A partitioned pipeline must use the {partition_seq} placeholder in its destination, so each partition gets its own file (batch-0.txt, batch-1.txt, ...).
Non-partitioned pipelines must not contain the placeholder.
partition is terminal — nothing can be chained after it.
partition() takes only a strategy: partition.count(n) or partition.items(n). A bare integer is rejected.
Execution is deterministic and order-preserving; an empty dataset produces zero partitions.

Pair partitioning with step.for_each to process each batch in parallel. The for_each items can be a typed transform expression directly:

BHW

Alternatively, write partitions to files first and fan out over them with path.glob(base = "batches", pattern = "scan-*.txt").

Records support field access (row.url, row.severity), method-style string ops on fields (row.host.lower()), and index access (row.urls[0]) inside map/filter lambdas and dataset mappings.