Anubis

Anubis is an open-source web AI firewall utility that filters incoming HTTP requests to protect upstream resources from scraper bots and AI crawlers. It operates by issuing one or more challenges to incoming connections to verify the nature of the request before allowing it to reach the origin server. This mechanism ensures that automated traffic is intercepted and validated before it can consume the resources of the backend infrastructure.

The software is deployed as a server application that sits in front of a website or API to act as a defensive layer. It is designed to be as lightweight as possible to ensure that the utility remains accessible for users with limited hardware resources. By intercepting traffic at the edge, it prevents the origin server from being overwhelmed by the high volume of requests typically generated by large scale AI data collection efforts.

Key features

• Filters incoming HTTP requests using challenges
• Protects upstream resources from AI crawlers
• Blocks scraper bots from accessing origin servers
• Supports configurable bot policy definitions
• Includes allowlist capabilities for specific bots
• Lightweight architecture for low resource overhead
• Verifies connections before forwarding to upstream resources

Anubis is intended for users who require a self-hosted security layer to prevent automated scraping when other proxy services are not an option. It allows administrators to define specific bot policy definitions to manage which entities are permitted to access the site. This granular control helps mitigate the risk of blocking beneficial services, such as the Internet Archive, by providing a way to explicitly allowlist known good bots. The architecture focuses on minimizing the footprint of the firewall while maximizing the difficulty for automated scrapers to bypass the challenge system.

This tool serves as a specialized security utility for protecting small internet communities and independent websites from aggressive automated data collection and AI scraping.