Search for a command to run...
I help companies to leverage Machine Learning to create innovative products through an end-to-end machine learning development process that designs, builds, and manages reproducible, testable, scalable, and evolvable ML-powered software with minimal cost.
No comments yet. Be the first to comment.
Here I show a summary description of the projects I've been working on in the last few years. Most of the projects have the source code publicly available.
This project contains a set of scripts used to scrape Ebay's products data using Scrapy Web Crawling Framework.
How AI-Driven Programming is Transforming the Tech Industry and Reshaping Developer Roles
How Automated Tests Can Transform Your Nights From Stressful to Restful
Streamlining Business Communication for Enhanced Collaboration, Customer Support, Marketing, and Decision-Making.
Machine Learning Operations is a process of automating and productionalizing machine learning applications and workflows.
In this project, I've built a Computer Vision model to find basket icons in e-commerces.
This project was built using Scrapy (Scraping and Web Crawling Framework).
It contains a set of Spiders to gather product data from Etsy Website.
The client needs data from thousands of products of etsy.com to perform data analysis.
Create an automated and fast solution to navigate the website, find the products by a search text, extract all the data, and save it in a user-friendly format (CSV and XLSX).
I've used the Scrapy Web Crawling Framework to build a Python script to search and scrape (extract) the data of products found in Etsy. Data collected: product_id, URL, price, rating, number_of_reviews, product_options, count_of_images, images_urls, favorited_by, store_name, and description.
The client was able to quickly download the data in CSV and Excel format of more than 100,000 products from etsy.com.
The data was used for data analysis and add great value to the client business.
The solution is available at Github.
You will need Python 3.6+ to run the scripts. Python can be downloaded here.
You have to install the Scrapy framework and other required packages:
pip install -r requirments.txtOnce you have installed the Scrapy framework, just clone/download this project:
git clone https://github.com/cpatrickalves/scraping-etsy
This Spider access the Etsy website and search for products based on a given search string.
Supported parameters:
For example, to search for '3d printed' products go to the project's folder and run:
scrapy crawl search_products -a search='3d printed'
To save the results, use -o parameter:
scrapy crawl search_products -a search='3d printed' -o products.csv
The Spider will create CSV and Excel files.
To limit the number of products scraped, use the count_max parameter:
scrapy crawl search_products -a search='3d printed' -a count_max=10 -o products.csv
The product reviews data can be obtained in three ways:
To choose the option to scraping the reviews use the -a reviews_option parameter:
scrapy crawl search_products -a search='3d printed' -a reviews_option=3 -o products.csv
You can change the number of concurrent requests performed by Scrapy in the setting.py file.
CONCURRENT_REQUESTS = 10
Change this if you want to decrease the number of requests to avoid get blocking by Etsy.
If you only need the products URLs, the scraping can be faster, just use the urls_only flag:
scrapy crawl search_products -a search='xbox controller elite' -o products.csv -a urls_only=true