Web Scraping in Node.JS: 5 Steps to Scrape a Sneakers Auction

You might have heard about web scraping and sneakers. You probably own some sneakers but don’t know as much about web scraping.

One of the ways people like to use web scraping is to get the best prices on rare and expensive sneakers. It just so happens that these kinds of projects are also a great way to learn more about web scraping as a whole and what it can achieve.

In this article, I want to combine a web scraping tool and an auction site to show you how web scraping can be used. Along the way, maybe we’ll get a good deal on a pair of kicks.

· What is web scraping?
· When is web scraping useful?
· What are the most important features of a web scraping tool?
· How does a web scraping project look like?
∘ Step 1: Make sure you have a WebScrapingAPI account
∘ Step 2: Install the dependencies
∘ Step 3: Scrape the page contents
∘ Step 4: Extract the information
∘ Step 5: Run the code
· Data extraction is as useful as you make it

What is web scraping?

Web scraping is a data scraping technique used to extract information from websites.

This can be done manually by going to the website and using copy-paste to get the information you need or automatically, by using a web scraping tool.

We’ve been hearing rumors for some time that the robots will take our jobs. Web scraping is the equivalent of a robot with a copy-paste feature. In this case, though, it’s more about robots making your job easier, not taking it.

When is web scraping useful?

Imagine working for a company where you need to keep updated a list with 100 rare products. The product’s price is listed on a different website.

Your morning routine would consist of going on each website, checking the price on the product page, and updating the price in the spreadsheet if anything happens to change. Sounds tedious and I’m certain you could do something more productive with your time.

Web scraping can replace all this repetitive and mundane work with an automatic process:

• Go on the website
• Get the product title
• Get the product price
• Display or store the data

That’s just one example. There are plenty of other situations where data extraction tools are just as useful, if not more so.

What are the most important features of a web scraping tool?

A web scraping tool is your pre-configured-ready-to-go virtual browser that can load any site you want and return its content in JSON or HTML formats.

On top of that, a web scraping tool fixes a lot of problems you will encounter if you want to collect data en masse:

• your IP address will be blocked because you are sending too many requests; you’ll need quality proxy services, such as rotating and residential IPs.
• some websites will not render if your browser doesn’t have Javascript enabled (curl, axios, got, request don’t); you will need to use a headless Chrome browser.
• geo-blocking is still a thing; using a web scraping tool, you teleport your browser to a different part of the world with the use of a globe-spanning proxy pool.

How does a web scraping project look like?

Let’s get back to sneakers. We’ll use the auction site Sotheby’s to look for any ongoing sneaker auctions. If we find any, we want to get the name and price. Once the code is done, you can run it at any point to check for any updates like new auctions or price changes.

The steps for scraping and parsing an auction website are simple:

1. Make sure you are using a web scraping service. I like to use WebScrapingAPI (the free account comes with 1000 requests).
2. Use WebScrapingAPI to scrape the auction website search results page; we will use axios for this
3. Parse the results and extract the products list; we will use cheerio just because it has the good old jQuery syntax
4. Store the product title, product price, and product price currency

So enough with the theory. It’s time to scrape the prices from a sneakers auction site in 5 easy steps with Node.JS.

Step 1: Make sure you have a WebScrapingAPI account

Just go to the WebScrapingAPI register page. It takes a few clicks to make an account and it comes with 1,000 free calls every month.

Step 2: Install the dependencies

We will use axios to connect to the WebScrapingAPI

npm install cheerio axios — save

Step 3: Scrape the page contents

The following code uses the axios library to access WebScrapingAPI and web scrape Sotheby’s search results for sneakers’ auctions. The result is returned in HTML format.

const cheerio = require(‘cheerio’);
const axios = require(‘axios’);
// Don’t forget to set the API key
const api_key = ‘ENTER_YOUR_API_KEY’;
// This points to the Sotheby’s sneakers page
const target_url = ‘https://www.sothebys.com/en/buy/sneakers?sliderPricePosLow=10000&sliderPricePosHigh=25000';
const api_url = `https://api.webscrapingapi.com/v1?api_key=${api_key}&url=${encodeURIComponent(target_url)}`;
(async () => {
 let response;
 try {
 response = await axios.get(api_url);
 } catch (error) {
 console.log(error);
 process.exit();
 }
 // Process results
 console.log(response.data);
 process.exit();
})();

This piece of code will use the WebScrapingAPI infrastructure to scrape the search results for sneakers from the Sotheby’s auction website.

Step 4: Extract the information

Now that we have the search results, we can extract only what we need:

• Product title
• Currency
• Price

Replace the //Process resultscomment with the following code, explained line by line:

// Convert the response to a Cheerio instance
const $ = cheerio.load(response.data);
// Get the products list
const $products = $(‘ul.gridContent > li’);
// Used to store the products info
const info = [];
// For each product
$products.each((index, product) => {
 // Get the product object
 const $product = $(product);
 // Get the price data
 const priceData = $product.find(‘[class^=”grid_card_listPrice”]’).text().trim().split(‘ ‘)
 // Store the product data
 info.push({
 title: $product.find(‘[class^=”grid_card_title”]’).text().trim(),
 price: parseFloat(priceData[0].replace(/,/ig, ‘’)),
currency: priceData[1],
 })
})

Step 5: Run the code

You can run the code with the following command:

node index.js

The output should look like this:

[
{
title: “Nike Air Jordan IV Retro OVO ‘Splattered’”,
price: 18850,
currency: ‘USD’
},
{
title: ‘Nike Air Force 1 Low Ueno Sakura’,
price: 13500,
currency: ‘USD’
},
{
title: “Nike Air Force 1 Hyperstrike Scarr’s Pizza”,
price: 12500,
currency: ‘USD’
},
{
title: ‘Nike Air Force 1 Promo Terror Squad’,
price: 13000,
currency: ‘USD’
},
{
title: ‘Nike Air Force 1 Supreme Quickstrike Year of the Rabbit’,
price: 13000,
currency: ‘USD’
},
{
title: ‘Nike Air Force 1 Hyperstrike 1World CLOT’,
price: 23000,
currency: ‘USD’
},
{
title: “Converse Dennis Rodman Game Worn All Star ‘AS Rodman’ Player Sample”,
price: 25000,
currency: ‘USD’
},
{
title: ‘Nike Scottie Pippen Game Worn Dual Signed Air Max Uptempo III Player Exclusive Sample’,
price: 15000,
currency: ‘USD’
},
{
title: ‘Nike Dunk Low Pro SB ‘What the Dunk’ Designed by James Arizumi 滑板鞋, 設計’,
price: 13000,
currency: ‘USD’
},
{
title: ‘Nike Tinker Hatfield Signed Air Max 1’,
price: 10000,
currency: ‘USD’
}
]

Data extraction is as useful as you make it

This was a quick overview of what web scraping is and how you can use it for your pet project or, if you’re brave enough, your next venture.

Here are a few things that you could do if you want to take your project to the next level:

1. Add support for more than one website
2. Group the products by name and implement multiple prices
3. Send the prices to email or post them on a public channel on Slack/Discord
4. Offer this functionality in return for a $1 recurring subscription and get closer to making your first million

One thing to keep in mind: a web scraping tool could save you weeks of work, time in which you can focus on the business logic and not the framework you want to build your business with.

You don’t have to take my word for this. Decide by yourself by investing 7 minutes of your time in this read to save a few weeks.