Crawler Pipeline
Producer → Queue → Consumer
Keywords & Seed URLs
Keywords (one per line) or Seed URLs
Workers
Discovery Workers
3
Processor Workers
2
Discovery Only
Processor Only
Pre-fetch HTML
Discovery Settings
Wave Size
Max Queue Depth
Max Discovered URLs
Processor Settings
Batch Size
Quality Threshold (0-100)
Relevance Threshold (0-100)
Max Imports
Search Engines
SearXNG
Google
Bing
Bing Regional
DuckDuckGo
Yahoo
Yandex
Brave
Wikipedia
Baidu
Direct Sites
Filters
Include Terms (one per line)
Exclude Terms (one per line)
Forced Domains (one per line)
URL Pattern (regex)
Advanced
Links per Page
Max Depth
Fetch Delay (ms)
0ms
HTTP Auth User
HTTP Auth Pass
Follow External
Posts First
Pagination
Launch Pipeline
Cleanup Old Data
Pause All
Resume All
Stop All
|
Back to Config
Export CSV
Export JSON
🔍
Discovery
idle
0
Discovered
0
Queued
0
Workers
0
URLs/s
📦
Queue
0
pending
⚙
Processor
idle
0
Imported
0
Processed
0
Duplicates
0
Errors
0
Workers
0
Avg ms
Rate Comparison (Discovery vs Import)
Queue Depth Over Time
Activity Log