crawling-and-serving/README.md

19 lines
488 B
Markdown
Raw Normal View History

2023-04-17 16:51:00 +00:00
# Crawing & Serving #
The crawler is a simple crawler that crawls the web and stores the results in a database and assets in a file system. The server is a simple server that serves the results of the crawler.
## Crawler ##
### Usage ###
Post a JSON object to the crawler with the following format:
`domain.com/crawl`
{
"url": "http://www.example.com",
}
The crawler will then crawl the given url and store the results in a database and assets in a file system.