Crawing & Serving

The crawler is a simple crawler that crawls the web and stores the results in a database and assets in a file system. The server is a simple server that serves the results of the crawler.

Crawler

Usage

Post a JSON object to the crawler with the following format:

domain.com/crawl { "url": "http://www.example.com", }

The crawler will then crawl the given url and store the results in a database and assets in a file system crawler_assests/www.example.com/.

API

The API is a simple API that serves the results of the crawler.

Routes

GET

/sites - Returns a list of all sites
/sites/:id - Returns the site object for the given site Id
sites/domain/:domain - Returns the domain object for the given domain

DELETE

/sites/:id - Deletes the site object for the given site Id
sites/domain/:domain - Deletes the domain object for the given domain

Post

sites/:id - Updates the site object for the given site Id

1 KiB Raw Blame History