Forage

Appearance

ForageA declarative DSL for data gathering

Write a recipe to gather structured records, enrich them against other sources, then share it on the hub.

DownloadLearn more
Forage

How it works ​

1
Gather
A recipe pulls typed records from any site or API. No bespoke code per source.
→
2
Enrich
Compose recipes to combine sources, and align fields to shared vocabularies like schema.org and Wikidata.
→
3
Share
Publish recipes and their types to the hub, where anyone can run, fork, or version them.

Example ​

Top 30 Hacker News front-page stories from the Algolia API. One step, no auth, no pagination.

forage
recipe "hacker-news"

type Story {
    title:    String
    url:      String?
    points:   Int
    author:   String
    comments: Int
}

step front {
    method "GET"
    url    "https://hn.algolia.com/api/v1/search?tags=front_page&hitsPerPage=30"
}

for $story in $front.hits[*] {
    emit Story {
        title    ← $story.title
        url      ← $story.url
        points   ← $story.points
        author   ← $story.author
        comments ← $story.num_comments
    }
}
sh
forage run hacker-news
json
{
  "observedAt": "2026-05-11T15:11:11Z",
  "records": [
    {
      "_typeName": "Story",
      "fields": {
        "title": "Hardware Attestation as Monopoly Enabler",
        "url": "https://grapheneos.social/@GrapheneOS/116550899908879585",
        "points": 1879,
        "author": "ChuckMcM",
        "comments": 617
      }
    },
    {
      "_typeName": "Story",
      "fields": {
        "title": "Local AI needs to be the norm",
        "url": "https://unix.foo/posts/local-ai-needs-to-be-norm/",
        "points": 1498,
        "author": "cylo",
        "comments": 580
      }
    }
  ]
}

Features ​

Recipes are data

A recipe describes what to gather. The engine is the only thing that runs HTTP or emits records.

Search and share

Publish a recipe; others browse, import, and version it instead of copy-pasting.

Two engines

Static reads the data straight from the response. Dynamic drives a real browser for sites that render it with JS, which also clears bot gates.

Live progress

Stream live status, requests sent, and the current URL into a UI or a log line.

Understand failures

Every run returns a structured report. Why it stopped, which rules never fired, which expectations didn't hold.

Assert expectations

Recipes declare their own coverage invariants ("at least 500 Products"). The engine checks them and reports gaps.

Archive and replay

Every run is archived to disk. Replay against the captures to iterate on extraction. No network needed.

Hot-reload

Edit a recipe and it reloads on save. A failed reload keeps the previous version live.