FLOWB0T
NEXUS v1.0.0
Dashboard
New Job
All Jobs
Results
Exports
Logs
Settings
System Online
Create New Crawl Job
Configure and start a new web crawling job
Back to Jobs
Job Information
Job Name
*
Give your job a memorable name for easy identification
Description
Job Type
Search-based (use search terms)
Domain-focused (target specific sites)
Direct URLs (crawl specific pages)
Mixed (combination)
Search Terms / Keywords
0 terms
Enter search terms (one per line)
Each term will search across Bing for web pages, images, and videos
Upload TXT File
Clear
Target Domains (Optional)
Specific domains to crawl
Leave empty to crawl the entire web. Add domains to focus crawl.
Direct URLs
URLs to crawl directly
Enter full URLs to crawl directly (one per line)
Filter Settings
Relevance Threshold
Min Content Length
Languages
English
Portuguese
Spanish
French
German
Blocked Domains (Blacklist)
Crawler Settings
Max Depth
Max Pages
Concurrent Connections
Request Timeout (seconds)
Delay Between (seconds)
Search Settings
Search Provider
Bing
Google (coming soon)
DuckDuckGo (coming soon)
Pages per Term
Search Types
Web
Images
Videos
Import Settings
Auto-import to database
Import to pinfeeds table
Create user accounts for authors
Deduplication
By URL
By Title
By Content Hash
All methods
Start Crawl Job
Save as Draft
Cancel
Estimated Statistics
~0
Est. URLs
~0h
Est. Time