The client needs data from thousands of equipment of National Instruments corporation type from testmart.com to perform data analysis.
Create an automated and fast solution to navigate the website, extract all the data of the National Instruments corporation type, and save it in a user-friendly format (CSV, JSON, and XML).
I've used the Scrapy Web Crawling Framework to build a bot to scrape (extract) the data of all equipment found in the National Instruments corporation type.
The client was able to quickly download the data of almost 50,000 different types of equipment from testmart.com.
The data was used for data analysis and add great value to the client business.
The solution is available at Github.
How to use
You will need Python 3.5+ to run the scripts. Python can be downloaded here.
You have to install the Scrapy framework and other required packages:
In command prompt/Terminal:
pip install -r requirements.txt
Once you have installed the Scrapy framework, just clone/download this project:
git clone https://github.com/cpatrickalves/scraping-gsamart.git
Access the folder in command prompt/Terminal and run the following command:
scrapy crawl gsamart_nic -o equipaments.csv
You can change the output format to JSON or XML by change the output file extension (ex: equipaments.json).