web.project package

Submodules

web.project.csv_database module

web.project.prod_extract module

web.project.web_crawler module

Web Crawler designed to find products and ratings for products developed and targeting seniors.

class project.web_crawler.WebCrawler(url=None, about=None, sub_url=None, page=None, data=None, clean=False)

Bases: object

Web Crawler

cleanup()

Clean up csv files in the current directory, and saves them to csv folder. Returns:

self.clean: bool - file cleaned.

compress()

Compresses files received from web-crawler.

csv_to_database()

Returns extracted csv data to an SQL database. Returns:

self.clean: bool - file cleaned.

data_extract()

Extract the url page data and parses the information with BeautifulSoup

get_data()

Get the data that the webcrawler is parsing Returns:

self.data: string - page data.

Example:

>>> example_data = crawler.get_data()

get_description()

Get the description of the product, located within the targeted web page. Returns:

self.about: string - description of product.

Example:

>>> example_description = crawler.get_description()

get_nav_categories()

Get the categories parsed within the webcrawler. Returns:

self.categories: list - list of categories within the navigation bar.

Example:

>>> example_categories = crawler.get_nav_categories()

get_nav_catlinks()

Get the category links within the webcrawler. Returns:

self.catlinks: list - list of category links within the navigation bar.

Example:

>>> example_catlinks = crawler.get_nav_catlinks()

get_page()

Gets the page that the webcrawler is parsing data from. Returns:

self.page: string - the page of the url.

Example:

>>> example_page = crawler.get_page()

get_sub_url()

Gets the url that the webcrawler will be accessing. Returns:

url: string - the url.

Example:

>>> example_url = crawler.get_url()

get_url()

Gets the url that the webcrawler will be accessing. Returns:

url: string - the url.

Example:

>>> example_url = crawler.get_url()

log_cleanup()

Clean up log files in the current directory, and saves them to log folder. Returns:

self.clean: bool - file cleaned.

open_log()

Creates log file for web-crawler.

sub_data_extract()

Extract the url page data and parses the information with BeautifulSoup

Module contents