Open Source Web Crawlers Software
Web Crawlers software automates the discovery and extraction of content from websites and social media platforms. These tools convert static pages into structured data or generate RSS feeds for sites that lack native syndication. This type of software is for researchers, archivists, and media collectors who need to automate the retrieval of videos, audio files, and metadata. Self-hosting these apps ensures that downloaded media and scraped datasets remain on local hardware without relying on external cloud storage or paying for API access.
This page lists 14 open source tools in the Web Crawlers category. The most popular are RSSHub, Maxun and VidBee. Most use the MIT or AGPL-3.0 license, and 13 offer an official Docker image.
RSSHub
Generate RSS feeds from various websites and social media platforms that do not provide them natively.
Maxun
A no-code platform to extract structured data from websites using AI, recording, and automated crawling tools.
VidBee
Download video and audio content from over 1000 websites with automated RSS feed subscription capabilities.
RSS-Bridge
Generates web feeds for websites that do not provide their own RSS or Atom feeds.
MediaGo
Download videos from YouTube and other platforms by sniffing m3u8 and HLS streams via desktop.
BiliTools
Download videos, audio, subtitles, and metadata from Bilibili using a lightweight cross-platform desktop interface.
Pinchflat
Self-hosted media manager that automatically downloads YouTube channels and playlists using yt-dlp for local archiving.
bitmagnet
A self-hosted BitTorrent indexer and DHT crawler with content classification and a web interface.
YaCy
A decentralized peer-to-peer search engine that allows users to host their own search index.
Wayback
A web archiving and playback tool that captures and preserves web content via an IM-style interface.
Fredy
A self-hosted real estate crawler that monitors German listing platforms and sends instant notifications for new properties.
Fess
An enterprise search server that crawls websites, file systems, and data stores for full text indexing.
Discount Bandit
Track product prices across multiple e-commerce stores and receive notifications when specific price criteria are met.
Kibitzr
A self hosted personal web assistant for automating routine tasks and monitoring web activity.
Join our newsletter to get shiny new open source software delivered to your inbox. Unsubscribe anytime.