kfir/crawling-and-serving
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

README.md

Browse source
This commit is contained in:
Kfir Dayan 2023-04-17 19:51:00 +03:00
parent 321a525e07
commit c5d5a6ac72
1 changed files with 19 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

20
README.md
View file

@ -1 +1,19 @@
# Crawing && Serving #
# Crawing & Serving #
The crawler is a simple crawler that crawls the web and stores the results in a database and assets in a file system. The server is a simple server that serves the results of the crawler.
## Crawler ##
### Usage ###
Post a JSON object to the crawler with the following format:
`domain.com/crawl`
{
"url": "http://www.example.com",
}
The crawler will then crawl the given url and store the results in a database and assets in a file system.