Forage

Appearance

Compose ​

Recipes emit typed records, so they fit together. compose chains recipes so one's output feeds the next, and an adapter recipe takes those records and adds fields, reshapes them, or reconciles them against another source. To give the resulting fields shared meaning, see Align.

Composition ​

A recipe's body is normally a sequence of step / for / emit statements. A composition body is a chain of recipe references instead:

forage
recipe "enriched-jobs"
emits EnhancedJobPosting

compose "scrape-job-board" | "to-enhanced"

The runtime runs scrape-job-board, then feeds the records it emits into to-enhanced as that recipe's input. A chain can be any length (A | B | C); each stage's output becomes the next stage's input.

For a stage boundary A | B to validate:

  • A emits exactly one type T, from its emits T clause or inferred from its emit statements.
  • B declares exactly one input slot typed [T] (B sees the whole stream at once) or T (B sees one record per upstream emission).

A recipe that composes itself, directly or through a cycle, is rejected: it would never terminate. References are plain strings, so any header name flows through, including hub packages: compose "@upstream/scrape-jobs" | "to-enhanced".

Adapter recipes ​

The common shape is an adapter: a recipe whose input is a parent type and whose output extends it.

forage
recipe "to-enhanced"
emits EnhancedJobPosting

share type JobPosting {
    id:    String
    title: String
}

share type EnhancedJobPosting extends JobPosting@v1 {
    salaryMin: Int?
    salaryMax: Int?
    remoteOk:  Bool
}

input postings: [JobPosting]

for $p in $input.postings[*] {
    emit EnhancedJobPosting {
        id        ← $p.id
        title     ← $p.title
        salaryMin ← null
        salaryMax ← null
        remoteOk  ← false
    }
}

EnhancedJobPosting extends JobPosting@v1 inherits the parent's fields and adds its own. The adapter rides the inherited fields through (id, title) and fills the new ones. Compose it onto any recipe that emits JobPosting:

forage
compose "scrape-job-board" | "to-enhanced"

@v1 is a version pin. extends @author/Name@v1 reaches a parent type published to the hub, so an adapter can extend someone else's type.

Reconciliation ​

An adapter can also pull from other sources mid-recipe, not just reshape what it was handed. The Wikidata pattern takes records carrying a wikidataId, fetches each entity, and merges selected claims:

forage
share type EnrichedCompany extends Company@v1 {
    founder:              String?
    headquartersLocation: String?
    inception:            String?
}

input companies: [Company]

for $c in $input.companies[*] {
    emit EnrichedCompany {
        id         ← $c.id
        name       ← $c.name
        wikidataId ← $c.wikidataId
        founder              ← wikidataEntity($c.wikidataId) | getField("P112")
        headquartersLocation ← wikidataEntity($c.wikidataId) | getField("P159")
        inception            ← wikidataEntity($c.wikidataId) | getField("P571")
    }
}

wikidataEntity(qid) is a transport-aware transform: it issues a request through the same engine transport your steps use, so --replay captures the reconciliation traffic alongside everything else. Field access goes through getField("P112") because a bare .field after a call is reserved for indexing in the grammar.

Discovering adapters on the hub ​

Because adapters extend shared types, the hub indexes both the parent and the child as first-class types. Its extends endpoint lists every adapter built on a given type, so an EnhancedJobPosting someone else published surfaces the moment you import JobPosting@v1. Composition works across authors, not just within a workspace. See Hub.

See also ​