Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About
The Top 10 Web Crawler Open Source Projects
Open source projects categorized as Web Crawler
Categories
>
Data Processing
>
Web Crawler
Edit Category
scrapy/scrapy
⭐
49,918
Scrapy, a fast high-level web crawling & scraping framework for Python.
dependent packages
0
total releases
0
most recent commit
about 2 years ago
huginn/huginn
⭐
40,328
Create agents that monitor and act on your behalf. Your agents are standing by!
dependent packages
0
total releases
0
most recent commit
about 2 years ago
dgtlmoon/changedetection.io
⭐
13,943
The best and simplest free open source website change detection, website watcher, restock monitor and notification service. Restock Monitor, change detection. Designed for simplicity - Simply monitor which websites had a text change for free. Free Open source web page change detection, Website defacement monitoring, Price change notification
dependent packages
0
total releases
0
most recent commit
about 2 years ago
apify/crawlee
⭐
11,229
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
dependent packages
0
total releases
0
most recent commit
about 2 years ago
crawlab-team/crawlab
⭐
10,521
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
dependent packages
0
total releases
0
most recent commit
over 2 years ago
ssssssss-team/spider-flow
⭐
8,075
新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。
dependent packages
0
total releases
0
most recent commit
almost 3 years ago
projectdiscovery/katana
⭐
7,995
A next-generation crawling and spidering framework.
dependent packages
0
total releases
0
most recent commit
about 2 years ago
lorien/awesome-web-scraping
⭐
6,060
List of libraries, tools and APIs for web scraping and data processing.
dependent packages
0
total releases
0
most recent commit
over 2 years ago
BruceDone/awesome-crawler
⭐
5,859
A collection of awesome web crawler,spider in different languages
dependent packages
0
total releases
0
most recent commit
over 2 years ago
pystardust/ani-cli
⭐
5,724
A cli tool to browse and play anime
dependent packages
0
total releases
0
most recent commit
about 2 years ago
Get A Weekly Email With Trending Web Crawler Projects
No Spam. Unsubscribe easily at any time.
Web Crawler
Subscribe
Javascript must be enabled to subscribe.
Privacy
|
About
|
Terms
|
Follow Us On Twitter
Copyright 2018-2026 Awesome Open Source. All rights reserved.
