Skip to content
meguri

Release notes

What changed in each meguri release.

The authoritative, commit-level history lives in CHANGELOG.md and on the releases page. This page summarises each version.

v0.2.0

The release where the .meguri file stops being a checkpoint you serialize and becomes the live store you ingest into and drain from directly, proven on the full Common Crawl at 2.08 billion URLs on one box.

  • The file is the live store. A frontier ingests, dedups, schedules, and recrawls straight against the file under bounded memory: URL strings spill to a disk arena, the seen-set and index live on disk as a DRUM, and an mmap reader serves a point lookup by URLKey, a sequential row cursor, and bulk load plus dedup. Residency tracks the working set, not the corpus, and a partition larger than memory checkpoints and recovers a region at a time without ever holding the whole thing on the heap.
  • Sharded across one box. Build and dedup fan out over hostkey ranges, and recrawl, compaction, and the schedule drain run K shards at a time, so a single machine spreads the frontier across partitions instead of one monolith.
  • The seed seam. A binary splittable .seed format and seedpack route a URL list into hostkey-sharded files through the reusable seed.ShardSet, the byte-for-byte format where a downloader and the store meet without a conversion. The Common Crawl downloader in ccrawl writes exactly this.
  • Smaller and faster on disk. The seen-set is a ribbon filter now, not a blocked Bloom, and a dedup hit trusts it and skips the base-confirm decode. The string blob is front-coded ahead of zstd, the shard key columns are paged so a confirm decodes one page not the whole column, and packed columns decode a word at a time. inspect and stats print the per-region byte breakdown so the cost of each part of the file is visible.
  • Published and reloaded from an open mirror. meguri export and meguri import move a partition to and from a Parquet dataset, and the scale harness measures the whole pipeline with per-stage IO faults, structured latency, and disk-index residency on a realistic multi-TLD corpus.
  • Proven at full scale. The whole CC-MAIN-2026-25 snapshot, 2.08 billion URLs, seeded at about 1.06M URLs/s into a 30.2 GiB .seed and built into a 256-partition, 52.89 GiB .meguri store at about 446k URLs/s, both under a few gigabytes of RSS.

v0.1.0

The first release. meguri is a distributed web-crawler frontier and rescheduler: it turns a stream of discovered links into a polite, freshness-aware crawl schedule and serializes the whole frontier to .meguri partition files a fleet redistributes by moving files.

  • A crawl frontier with all three forces. A URL is ordered by importance, released only when politeness allows (one in-flight fetch per host and per IP, spaced by a crawl delay), and rescheduled on a per-URL change-rate estimate so the crawl tracks the web instead of photographing it once. The three pull against each other and the engine resolves the conflict in that order.
  • A seen-set that holds work, not duplicates. Discoveries fold in through canonicalisation and an approximate membership filter, so a link already known is dropped before it reaches the queue. The filter serializes into the partition so a reload does not rebuild it from scratch.
  • The staged engine loop. meguri seed builds a frontier from a Common Crawl URL list; meguri run drains a checkpoint in priority-then-politeness order with the fetcher bound through a small interface; meguri serve opens a directory as a durable, log-structured partition that recovers on restart and checkpoints on shutdown.
  • The .meguri file. One partition serializes to one self-describing, columnar, checksummed file: a 64-byte header, the URL and host tables, the schedule index, the seen-set filter, and a string arena, all bracketed by the magic MEG1 and read from the tail in two small reads. Encoding is deterministic, so a checkpoint is diffable and a redistribution is verifiable byte for byte. meguri inspect, schedule, and stats read it; pack and compact write and consolidate it.
  • Fleet routing and projection. meguri map reads a fleet manifest and routes a host through the partition map; meguri bench measures the real per-partition bytes/url and seen-set bits/url on a corpus slice and projects the cost to a hundred billion URLs against the named scaling walls.
  • Pure Go, packaged everywhere. No cgo anywhere in the tree, CGO_ENABLED=0. The release ships archives for Linux, macOS, Windows, and FreeBSD on amd64 and arm64, .deb/.rpm/.apk packages, a multi-arch GHCR container image, checksums, SBOMs, a cosign signature, and Homebrew and Scoop entries, all from a single tag push.
4 min read