webcrawler/README.md

# webcrawler

## Instructions to Run 
1. Clone the repository
```
git clone https://github.com/AmanTahiliani/webcrawler.git
```
2. cd into the directory
```
cd webcrawler
```
3. Create a virtual environment
```
python3 -m venv venv
```
4. Activate the virtual environment
```
source venv/bin/activate
```
5. Install the requirements
```
pip install -r requirements.txt
```
6. Run the shell script
```
./amantahiliani.sh 
```

## Usage

Note: If you get a permission denied error, run the following command
```
chmod +x amantahiliani.sh
```
If it still doesn't work, run the following command
```
python3 MultiThreadedCrawler.py {seed_url} --pages_to_parse=1000
```


## Output
The output is generated in the form of the following files:
1. pages_per_second.png - A graph showing the number of pages crawled per second
2. craw_ratio_table.png-   A table showing the ratio of pages crawled to pages discovered
3. crawlspeed.png- A graph showing the crawl speed in pages per minute
4. Keywords_Output.json- A json file containing the keywords and the urls they were found in along with the frequency of the keyword in the url