- Introduced a new crawling management feature allowing users to configure, execute, and log web crawls.
- Added CRUD operations for crawl configurations, including URL analysis and preview capabilities.
- Implemented a new service for handling crawling logic and scheduling tasks.
- Integrated cheerio for HTML parsing and axios for HTTP requests.
- Created a sample HTML page for testing crawling functionality.
This commit enhances the application's data collection capabilities from external websites.
5da134f016