queue is full, can't add url (was fixed in 0.7.2)

[dynabler]

In issue #42 there was a limit on url queue of 1024, which was fixed for Flyscrape v0.7.2 to 1M. But somehow v0.9.0 has the same issue. I checked the code, but it's the same code as v0.7.2.

I'm not entirely sure, but could it be config depth, that it somehow leaks? spills over and goes rouge on other links? That way, you can most definitly get over 1M urls. My script had depth: 0, when this happenend and url text file has 4501 urls.

I also noticed my mistake, of adding concurrence to the script and the command-line. Could that be the problem? I just add it all as it happend, so you don't chase ghosts.

import urls from "./urls.txt"

export const config = {
  urls: urls.split("\n"),
  headers: {
    "User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36",
    "Cookie": 'cookie-settings={"all":0,"functional":0,"essentials":1,"analytical":0}'
  },
  depth: 0,
  concurrency: 1
};

export default function({ doc, absoluteURL, url, timestamp }) {
  const id = url
  const followers = doc.find('.header__follow-count').get(0);
  const fans = doc.find('.header__follow-count').get(1);
  const vids = doc.find('._title:contains("Videos")')
  const year = String(new Date().getFullYear());
  const title = doc.find('.header__title-text')

  return { 
    id: url.replace(/.+\/(.+)/,'$1'),
    stats: {
      followers: followers.text().trim(),
      fans: fans.text().trim()
    }
  }
}

and the command used:

    flyscrape run stats.js \
      --output.file "pages/stats_raw.json" \
      --output.format json \
      --rate 6 \
      --concurrency 1

[dynabler]

I think I finally found the problem causing this (with the help of AI):

TLDR: Flyscrape is faster than bbolt/sqlite read/write. Even if it's scraping a local web (saved pages from the internet and running a local web server for Flyscrape to get data), the read/write causes bbolt to crumble under load. So unlimited concurrency is, as the code sits, impossible (see example code below)

In the message above, I had concurrency 1 in the file and --concurrency 1 in the command. But I did get the queue is full. I'm thinking, it's a good idea to do write first, then add a new url, instead of keep adding urls and hoping bbolt accepts.

In web scrapers, bbolt "queue full" errors almost always point to an architectural design flaw in how the scraper handles worker concurrency alongside database writes.

Why Web Scrapers Trigger BBolt Queue Limits

High Worker Concurrency vs. Single-Writer Lock: If the scraper runs 30+ concurrent HTTP workers, they all attempt to mark URLs as "visited" or save scraped payloads to BBolt at the same time. Because BBolt permits only one write transaction at a time, workers queue up waiting for the lock until the scraper's internal buffer channel overflows.

Holding Transactions Across Network Calls: Poorly designed scrapers open a BBolt transaction to check if a URL is in the database, keep that transaction open while making the HTTP request over the internet, and only close it after writing. A slow or hanging network request holds the DB lock and starves all other workers.

Unbatched Disk Writes: If the scraper issues an individual db.Update() for every single page or link discovered, the disk forced-sync (fsync) overhead slows down writes to a crawl, creating a backlog in the scraper's queue.

For an embedded single-writer database like BBolt running on standard NVMe storage, the optimal scraper concurrency is usually between 8 and 16 concurrent workers.

Going higher rarely improves your overall throughput—it just shifts the bottleneck from the network to the database disk lock, causing worker queues to back up and crash.

Concurrency Level	DB Lock Contention	Scraping Behavior
4 to 8 Workers	Extremely Low	Rock Solid. Best for single-disk machines, low-RAM VPS setups, or scrapers doing heavy DB updates per page.
8 to 16 Workers	Moderate	The Sweet Spot. Maximum real-world page throughput without overloading single-writer lock queues.
20 to 32 Workers	High	Diminishing Returns. Workers spend more time waiting in line for db.Update() than actually fetching pages.
50+ Workers	Critical	Queue Failure. Triggers "Queue is Full" / ErrDatabaseFull errors unless writes are strictly batched in memory.

How to Maximize Throughput at 10–15 Workers

If you want the speed of 50 workers without crashing BBolt, structure the scraper around these two rules:

Decouple Fetching from Writing (In-Memory Queue)
Never open a BBolt transaction inside the fetch loop. Let 15 workers fetch HTML into an in-memory queue (like a Go channel), and use one single dedicated background worker to write results to BBolt.

Batch Writes in Memory
Instead of writing every single URL or page individually, buffer 100 to 500 items in memory and flush them to BBolt inside a single db.Update() block:

// Good Pattern: 1 transaction for 500 items = 1 disk fsync instead of 500
db.Update(func(tx *bolt.Tx) error {
    b := tx.Bucket([]byte("ScrapedData"))
    for _, item := range bufferedItems {
        b.Put([]byte(item.URL), []byte(item.HTML))
    }
    return nil
})

In other words, it's like redis, work from memory, instead of disk (memory is 10k faster than disk)