feat: add web crawling management functionality

- Introduced a new crawling management feature allowing users to configure, execute, and log web crawls.
- Added CRUD operations for crawl configurations, including URL analysis and preview capabilities.
- Implemented a new service for handling crawling logic and scheduling tasks.
- Integrated cheerio for HTML parsing and axios for HTTP requests.
- Created a sample HTML page for testing crawling functionality.

This commit enhances the application's data collection capabilities from external websites.
This commit is contained in:
kjs
2026-03-26 16:30:53 +09:00
parent 07777e314b
commit 5da134f016
8 changed files with 1700 additions and 0 deletions

View File

@@ -27,6 +27,7 @@
"axios": "^1.11.0",
"bcryptjs": "^2.4.3",
"bwip-js": "^4.8.0",
"cheerio": "^1.2.0",
"compression": "^1.7.4",
"cors": "^2.8.5",
"docx": "^9.5.1",
@@ -50,6 +51,7 @@
"nodemailer": "^6.10.1",
"oracledb": "^6.9.0",
"pg": "^8.16.3",
"playwright": "^1.58.2",
"quill": "^2.0.3",
"react-quill": "^2.0.0",
"redis": "^4.6.10",