34 lines
No EOL
1 KiB
Markdown
34 lines
No EOL
1 KiB
Markdown
# Crawing & Serving #
|
|
|
|
The crawler is a simple crawler that crawls the web and stores the results in a database and assets in a file system. The server is a simple server that serves the results of the crawler.
|
|
|
|
|
|
## Crawler ##
|
|
|
|
|
|
### Usage ###
|
|
Post a JSON object to the crawler with the following format:
|
|
|
|
`domain.com/crawl`
|
|
{
|
|
"url": "http://www.example.com",
|
|
}
|
|
|
|
The crawler will then crawl the given url and store the results in a database and assets in a file system
|
|
`crawler_assests/www.example.com/`.
|
|
|
|
|
|
# API #
|
|
|
|
The API is a simple API that serves the results of the crawler.
|
|
|
|
# Routes #
|
|
## GET ##
|
|
/sites - Returns a list of all sites
|
|
/sites/:id - Returns the site object for the given site Id
|
|
/sites/domain/:domain - Returns the domain object for the given domain
|
|
## DELETE ##
|
|
/sites/:id - Deletes the site object for the given site Id
|
|
/sites/domain/:domain - Deletes the domain object for the given domain
|
|
## Post ##
|
|
/sites/:id - Updates the site object for the given site Id |