feat: add web crawling management functionality
- Introduced a new crawling management feature allowing users to configure, execute, and log web crawls. - Added CRUD operations for crawl configurations, including URL analysis and preview capabilities. - Implemented a new service for handling crawling logic and scheduling tasks. - Integrated cheerio for HTML parsing and axios for HTTP requests. - Created a sample HTML page for testing crawling functionality. This commit enhances the application's data collection capabilities from external websites.
This commit is contained in:
@@ -27,6 +27,7 @@
|
||||
"axios": "^1.11.0",
|
||||
"bcryptjs": "^2.4.3",
|
||||
"bwip-js": "^4.8.0",
|
||||
"cheerio": "^1.2.0",
|
||||
"compression": "^1.7.4",
|
||||
"cors": "^2.8.5",
|
||||
"docx": "^9.5.1",
|
||||
@@ -50,6 +51,7 @@
|
||||
"nodemailer": "^6.10.1",
|
||||
"oracledb": "^6.9.0",
|
||||
"pg": "^8.16.3",
|
||||
"playwright": "^1.58.2",
|
||||
"quill": "^2.0.3",
|
||||
"react-quill": "^2.0.0",
|
||||
"redis": "^4.6.10",
|
||||
|
||||
Reference in New Issue
Block a user