Hello and welcome
This is the October 10 letter
Introduction
The story begins last month, when I was working on a small project, and the main idea was to create an AI Travel Itinerary app that helps people to find good places, along with some other details like itinerary and top Airbnb and history and culture
For creating an AI travel itinerary app, I integrate an AI LLM and provide a prompt input, but it seems that the AI doesn't have access to exact or latest Google Maps locations, trending Airbnbs and more real-time data, and that is where the idea of scraping came to me.
But before scraping content, I was also searching for a Kaggle dataset or github repo because I knew someone in the past would have gone through the same problem and have already created the same dataset. Unfortunately, I couldn't find one.
Web Scraping
Data scraping is the last option left, so I've to choose it because of 2 following two reasons
- No existing dataset can be found on the internet
- Scraping gives the power to LLM or AI to become more real-time
Before we get started, check the dataset on Kaggle, Indian cities, states, and places, locations and the Wikipedia dataset.
This dataset contains place names, latitude, longitude, state names, unsplash images, Google images, Wikipedia content and Airbnb listings for the specific place.
Since one can understand that for each data type we need to make our own scraping API, and for that I chose Hono as the new alternative to Express.js
Hono as Backend
Honestly, I am loving Hono; it's simple and lightweight and quickly provides a way to create APIs and can be hosted on Vercel as well.
To get started with hono, read this article
Installing Packages
A few packages apart from Hono we need are as follows for scraping
- Pupeteer core
- Cheerio
Both of them help in crawling Google and scraping the website works headless. But since I want to host my API on Vercel Edge, I've to use smaller size packages for the above
- puppeteer-core
- @sparticuz/chromium
Read more about @sparticuz/chromium as its lighter version and works on Vercel Edge on a Headless browser.
Scraping APIs
One common way to scrape Google images, unsplash images and wikipedia and Airbnb listings is to open a headless Chrome browser and open the URL using a query for each method, and then use Cheerio or custom parsing ways using classname or id to fetch the accurate content or scrape the accurate key values, such as images, listings, latitude and longitude
The first API I've created easily using Puppeteer is scraping Google Maps using a single query
The API use a headless Chrome browser, takes the user query, and Puppeteer opens the same URL and fetches the HTML
Google Maps Scraping API
// Google Maps scraping endpoint using headless Chrome
app.post("/scrap-google-maps", async (c) => {
try {
const { queries, singleQuery } = await c.req.json();
if (!queries && !singleQuery) {
return c.json(
{
success: false,
error: "Either 'queries' array or 'singleQuery' string is required",
},
400
);
}
// Handle both single query and an array of queries
const queryArray = Array.isArray(queries)
? queries
: [queries || singleQuery];
if (!queryArray.length || queryArray.some((q) => !q)) {
return c.json(
{
success: false,
error: "At least one valid query is needed",
},
400
);
}
let browser;
try {
// Launch headless Chrome browser with proper configuration
browser = await chromium.launch({
headless: true,
userAgent:
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36",
args: [
"--no-sandbox",
"--disable-setuid-sandbox",
"--disable-dev-shm-usage",
"--disable-accelerated-2d-canvas",
"--no-first-run",
"--no-zygote",
"--single-process",
"--disable-gpu",
],
});
// Create a shared context for all queries
const context = await browser.newContext();
// Block unnecessary resources to improve performance
await context.route("**/*", (route) => {
const type = route.request().resourceType();
return ["image", "font", "stylesheet"].includes(type)
? route.abort()
: route.continue();
});
// Process all queries in parallel using Promise.all
const results = await Promise.all(
queryArray.map(async (query) => {
const page = await context.newPage();
try {
// Navigate to Google Maps
await page.goto(
`https://www.google.com/maps/search/${encodeURIComponent(query)}`,
{
waitUntil: "networkidle",
timeout: 30000,
}
);
// Wait a bit for the location to be fully loaded
await page.waitForTimeout(5000);
const locationResults = await page.evaluate(() => {
// Select the results container with role feed and aria-label with Results for ...
const resultsContainer = Array.from(
document.querySelectorAll('div[role="feed"]')
).find((el) =>
el.getAttribute("aria-label")?.includes("Results for")
);
if (!resultsContainer) return [];
// Each child div under feed represents a location card
const cards = Array.from(
resultsContainer.querySelectorAll("div")
).slice(0, 10);
return cards.map((card) => {
// Name
const name =
card
.querySelector('[class*="fontHeadlineSmall"]')
?.textContent?.trim() || "";
// Rating text (e.g. "Rated 4.5 stars out of 5")
const ratingLabel =
card
.querySelector('[class*="fontBodyMedium"]')
?.getAttribute("aria-label") || "";
const ratingMatch = ratingLabel.match(/Rated ([0-9.]+) stars?/);
const rating = ratingMatch ? parseFloat(ratingMatch[1]) : null;
// Review count text (e.g. "200 reviews")
const reviewsText =
card
.querySelector('[aria-label$="reviews"]')
?.getAttribute("aria-label") || "";
const reviewsMatch = reviewsText.match(/(\d+,?\d*) reviews/);
const reviews = reviewsMatch
? reviewsMatch[1].replace(/,/g, "")
: null;
// Google Maps URL link
const url =
card.querySelector(
'a[href*="https://lh3.googleusercontent.com/gps-cs-s"]'
)?.href || "";
// Image URL (if exists)
const image = card.querySelector("img[src]")?.src || "";
return { name, rating, reviews, url, image };
});
});
return { query, ...locationResults };
} catch (error) {
console.error(`Error processing query "${query}":`, error);
return {
query,
name: "",
address: "",
coordinates: null,
url: "",
details: [],
rating: "Rating not available",
reviews: "Reviews not available",
type: "Location",
error: error.message,
};
} finally {
await page.close();
}
})
);
// Close the shared context after all queries are complete
await context.close();
// If single query was provided, return just the location data
if (!Array.isArray(queries)) {
const result = results[0];
if (!result.coordinates) {
return c.json(
{
success: false,
error: "Location not found or coordinates not available",
data: result,
},
404
);
}
return c.json({
success: true,
data: result,
});
}
// For multiple queries, return array of results
return c.json({
success: true,
data: {
totalQueries: queryArray.length,
results: results,
generatedAt: new Date().toISOString(),
},
});
} finally {
if (browser) {
await browser.close();
}
}
} catch (error) {
console.error("Google Maps Scraping Error:", error);
return c.json(
{
success: false,
error: "Failed to fetch location data",
details: error.message,
},
500
);
}
});In the API,
- First, we define a headless browser
- Then we define the URL using the user query
- Once the browser reached the URL, we optimised it by removing/disabling stylesheets, images
- Process all queries in parallel and get the results in an array
- Finally, return the array with scraped data containing the latitude and longitude
Please note - This API will not work in production, as you need proxies within the API to keep changing the user agent so that Google won't trace your identity and block you. This prevents the blocking by Google in production.
For proxies, I was using the Brightdata website, but one can use other resources as well.
Other APIs
Once we get the cities from the Indian cities JSON github file, I use this to store each of the cities' latitude and longitude and other details using the Google Maps scrap API
Then we use the same data column to fetch Airbnb listings, the city and state name and process the bulk Airbnb listings scrape
Similarly, I did with Wikipedia, Google images and unsplash images.
Scrap Airbnb Listings API
app.post("/scrap-airbnb", async (c) => {
const {
city,
state,
country = "India",
checkin,
checkout,
useProxy = false,
} = await c.req.json();
if (!city || !state) {
return c.json({ error: "City and state are required" }, 400);
}
// Generate Airbnb search URL
const searchQuery = `${city.replaceAll(" ", "-")}--${state.replaceAll(
" ",
"-"
)}`;
const airbnbUrl = `https://www.airbnb.com/s/${searchQuery}--${country}/homes`;
// Add query parameters if provided
const urlParams = new URLSearchParams();
if (checkin) urlParams.append("checkin", checkin);
if (checkout) urlParams.append("checkout", checkout);
const fullUrl = urlParams.toString()
? `${airbnbUrl}?${urlParams.toString()}`
: airbnbUrl;
console.log(`🏠 Scraping Airbnb listings from: ${fullUrl}`);
let scrapedData = {};
try {
const response = await fetch(`http://localhost:3001/scrap-url`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
url: fullUrl,
useProxy: false,
}),
});
if (!response.ok) {
throw new Error(`HTTP error! status: ${response.status}`);
}
const scrapData = await response.json();
const allLinks = scrapData.data.links;
const listings = [];
// Process each listing link
const elementsToProcess = allLinks.length;
for (let i = 0; i < elementsToProcess; i++) {
const link = allLinks[i];
try {
// Extract listing URL
const listingUrl = link.href || link.url || "N/A";
// Only include links that match the required Airbnb room URL pattern
if (
typeof listingUrl === "string" &&
listingUrl.startsWith("https://www.airbnb.co.in/rooms")
) {
// Extract title from link text
const title = link.text ? link.text : link.title;
// Create listing data object
const listingData = {
title: title,
url: listingUrl,
};
listings.push(listingData);
}
} catch (error) {
console.warn(`Error processing listing ${i}:`, error);
continue;
}
}
// Get page info from scrap-url response
const pageInfo = {
title:
scrapData.data?.pageInfo?.title ||
scrapData.data?.title ||
scrapData.title ||
"Airbnb Search Results",
url: fullUrl,
description:
scrapData.data?.pageInfo?.description ||
scrapData.data?.description ||
scrapData.description ||
"",
};
scrapedData = {
success: true,
url: fullUrl,
searchQuery: searchQuery,
checkin: checkin || null,
checkout: checkout || null,
listings: listings,
totalListings: listings.length,
pageInfo: pageInfo,
timestamp: new Date().toISOString(),
useProxy: useProxy,
};
} catch (error) {
console.error("❌ Error scraping Airbnb:", error);
scrapedData = {
success: false,
error: error.message,
url: fullUrl,
timestamp: new Date().toISOString(),
};
}
return c.json(scrapedData);
});In the Airbnb scraping API,
- We are simply using the city, state other details from the user query to create a search URL
- Then use a headless browser to reach the URL
- Fetch the HTML and then run our own parser method to scrape listings
- Finally, return all listings for the user query
Scrap Wikipedia API
async function checkWikipediaURL(url) {
try {
const res = await fetch(url, { method: "HEAD" });
return res.ok;
} catch {
return false;
}
}
// Wikipedia URL generation utility with 3 specific strategies
function generateWikipediaURLs(cityName, stateName) {
const urls = [];
// Clean city name - replace spaces with underscores
const cleanCity = cityName.trim().replace(/\s+/g, "_");
// Clean state name - keep original spaces, don't add underscores
const cleanState = stateName ? stateName.trim() : "";
// Strategy 1: domain.com/wiki/city,_state (with original spaces in state)
if (cleanState) {
urls.push(`https://en.wikipedia.org/wiki/${cleanCity},_${cleanState}`);
}
// Strategy 2: domain.com/wiki/city,_state (with underscores in state)
if (cleanState) {
const cleanStateWithUnderscores = cleanState.replace(/\s+/g, "_");
urls.push(
`https://en.wikipedia.org/wiki/${cleanCity},_${cleanStateWithUnderscores}`
);
}
// Strategy 3: domain.com/wiki/city (city only, no state)
urls.push(`https://en.wikipedia.org/wiki/${cleanCity}`);
return urls;
}
// Wikipedia content extraction utility using existing scrap-url API
async function extractWikipediaContent(wikipediaUrl, useProxy = false) {
try {
console.log(`🔍 Scraping Wikipedia content from: ${wikipediaUrl}`);
// Use the existing scrap-url API endpoint
const response = await fetch(`http://localhost:3001/scrap-url`, {
method: "POST",
headers: {
"Content-Type": "application/json",
},
body: JSON.stringify({
url: wikipediaUrl,
useProxy: useProxy,
includeImages: false,
includeLinks: false,
extractMetadata: true,
timeout: 30000,
waitForSelector: "#mw-content-text",
}),
});
if (!response.ok) {
throw new Error(`HTTP error! status: ${response.status}`);
}
const scrapedData = await response.json();
if (!scrapedData.success) {
return {
success: false,
error: scrapedData.error || "Failed to scrape Wikipedia page",
url: wikipediaUrl,
};
}
// Extract paragraphs from semantic content
const paragraphs =
scrapedData.data?.content?.semanticContent?.paragraphs || [];
if (paragraphs.length === 0) {
return {
success: false,
error: "No paragraphs found in Wikipedia content",
url: wikipediaUrl,
};
}
return {
success: true,
data: {
title: scrapedData.data.title,
url: wikipediaUrl,
extractedAt: new Date().toISOString(),
paragraphs: paragraphs,
summary: paragraphs.slice(0, 3).join(" ").substring(0, 500) + "...",
rawData: scrapedData.data,
},
url: wikipediaUrl,
};
} catch (error) {
console.error(
`❌ Wikipedia content extraction error for ${wikipediaUrl}:`,
error
);
return {
success: false,
error: error.message,
url: wikipediaUrl,
};
}
}
In the above API, I've added 2 methods
- First method: check a valid Wikipedia URL
- The second one, generateWikipediaURLs, creates an accurate URL using the name of the city or state
- And the last method simply fetches the content from the HTML we got by making an API request to the valid Wikipedia URL
Sometimes, each platform makes changes in its HTML DOM structure to make sure those bots and scrapers keep failing from time to time, and in that case, proxies won't work well and nor will existing code. The only way to work is to keep iterating code and find one solid way to deal with scraping accurate content from the web URL or HTML file
Bulk Scraping
This is quite a tricky and time-consuming task, since I've 1000+ places or cities, I've to start the parallel processing, meaning 3 max requests of the headless browser will run at the same time, this will consume CPU resources, but it will speed up the entire bulk scraping process.
One example, as shown below how we are using a bulk scraping process for Wikipedia content
// Background processing function
async function processWikipediaBatches(
batches,
maxConcurrentBrowsers,
useProxy
) {
try {
// Process batches sequentially to avoid overwhelming Wikipedia
for (let batchIndex = 0; batchIndex < batches.length; batchIndex++) {
const { batchNumber, locations } = batches[batchIndex];
// Mark batch as started
wikipediaScrapingQueue.markBatchStarted(batchNumber);
// Process current batch
const batchResults = await processWikipediaBatch(
locations,
maxConcurrentBrowsers,
useProxy
);
// Mark batch as completed
wikipediaScrapingQueue.markBatchCompleted(batchNumber, batchResults);
// Store results in Supabase
await storeWikipediaResults(batchResults);
// Add delay between batches to be respectful to Wikipedia
if (batchIndex < batches.length - 1) {
const delay = Math.min(5000, locations.length * 100); // Adaptive delay
console.log(`⏳ Waiting ${delay}ms before next batch...`);
await new Promise((resolve) => setTimeout(resolve, delay));
}
}
console.log("🎉 All Wikipedia batches completed successfully!");
wikipediaScrapingQueue.isProcessing = false;
} catch (error) {
console.error("❌ Error processing Wikipedia batches:", error);
wikipediaScrapingQueue.isProcessing = false;
}
}The same bulk parallel processing is used for scraping Google Maps, Airbnb listings and more.
Lastly, do check the dataset on Kaggle, Indian cities, states, and places, locations and the Wikipedia dataset
That's it for today, see you in the next one
Shrey
iHateReading