CrawlPy is the perfect Python library for easy and efficient web crawling. It's designed to be simple yet powerful, making it easy to fetch and process web content for a variety of tasks.
- Minimalist Design: Lightweight and uncluttered, focusing on the essentials of web crawling without unnecessary complexity.
- Effortless API: Intuitive API for making HTTP requests and processing responses easily.
- Flexibility: Adaptable for various needs, whether you're scraping data, monitoring websites, or conducting research.
- Customization: Full control over your requests with support for cookies and custom headers.
You can start web crawling with CrawlPy in just a few simple steps:
-
Installation
Install CrawlPy via pip:
pip install crawlpy
-
Write Your Crawler
Create a Python file for your crawler, like
crawler.py
:import asyncio from crawlpy import CrawlPy # Create a CrawlPy object crawler = CrawlPy() url = "http://example.com" # Get the event loop loop = asyncio.get_event_loop() try: # Fetch the HTML content asynchronously html_content = loop.run_until_complete(crawler.get(url)) print(html_content) finally: # Close the crawler session loop.run_until_complete(crawler.close()) # Close the event loop loop.close()
-
Run Your Crawler
Run your crawler script:
python crawler.py
And watch CrawlPy fetch web content effortlessly!
CrawlPy is licensed under the CC0-1.0 License. For details, see LICENSE.