crawling-and-serving
Akamai Task
| src | ||
| test | ||
| .eslintrc.js | ||
| .gitignore | ||
| .prettierrc | ||
| docker-compose.yaml | ||
| Dockerfile | ||
| nest-cli.json | ||
| package-lock.json | ||
| package.json | ||
| README.md | ||
| tsconfig.build.json | ||
| tsconfig.json | ||
Crawing & Serving
The crawler is a simple crawler that crawls the web and stores the results in a database and assets in a file system. The server is a simple server that serves the results of the crawler.
Crawler
Usage
Post a JSON object to the crawler with the following format:
domain.com/crawl
{
"url": "http://www.example.com",
}
The crawler will then crawl the given url and store the results in a database and assets in a file system
crawler_assests/www.example.com/.