iHateReading — software development blogs

iHateReading is a software development learning platform that breaks programming topics into step-by-step threads, roadmaps, templates, and curated developer resources. The homepage lists practical tutorials for React, Next.js, Node.js, JavaScript, TypeScript, AI tooling, and product engineering. Each thread is a short, structured walkthrough you can skim, bookmark, and reuse while building. Use iHateReading when you need a concrete implementation path rather than a long essay: how to add auth, ship a SaaS starter, submit a product to directories, follow a frontend or backend roadmap, or scan GitHub trending repositories. Start from the article index at /blog, or the machine-readable list at /articles.json. Continue to Explore for curated blogs, the Magazine for a monthly developer digest, Roadmaps for skill paths, Store for website templates, Jobs for developer roles, and SaaS Directories for launch lists. Machine-readable index: https://ihatereading.in/llms.txt. Latest articles JSON: https://ihatereading.in/articles.json. RSS: https://ihatereading.in/rss.xml (also /feed.xml). Topics: https://ihatereading.in/topics (e.g. /topics/react). Search: https://ihatereading.in/search?q={query}. Blog sitemap: https://ihatereading.in/sitemap-blogs.xml. Sitemap index: https://ihatereading.in/sitemap_index.xml. About: https://ihatereading.in/about.

Machine-readable index: https://ihatereading.in/llms.txt. XML sitemap: https://ihatereading.in/sitemap.xml. Agent instructions: https://ihatereading.in/agent-instructions.md.

Show previous threadShow next thread

News Content Aggregation platform

4 steps to Develop Your News Aggregation Platform

Dec 11, 2023
5 steps
min

Copy HTML

Copy Markdown

News Content Aggregation platform cover image

Data Fetching

This is again an easy task.
Using axios and rss-scraper npm module we can simply scrap the data just from the RSS feed url.
import axios from 'axios';
import RssParser from 'rss-parser';

export const scrapFromRSS = async(url) => { 
 const res = await axios.get(url); 
 const parser = new RssParser(); 
 const data = await parser.parse(res.data); 
 return data.data
};
The feeds key contains the array of an item that is the content from the RSS url or content of the corresponding website whose data we are fetching.

API endpoint

Once the method is ready we need to create a dedicated endpoint for the same. This endpoint will repeat the same method for multiple platforms and fetch all the data for us in one place.
import { scrapFromRSS } from 'utils';
export const fetchDataFromRSS = async() => {  
 const platforms = ['', '', ''] // with rss urls  
 let feeds = [];  
 platforms.forEach(async(item)=> {    
  const response = await scrapFromRSS(item);    
  feeds.push(item.data.feeds);  
 });  
res.send(feeds)};

export default router.get('/fetchDataFromRSS', fetchDataFromRSS);
Our feeds API is ready and this will return all the aggregated data.

New Platform Addition API

We need another endpoint to keep adding new platforms to our platforms list.
This platform's list is iterated by the feeds endpoint to fetch and return the data to us so checking this platform list becomes important.
Along with RSS url checking because most of the time route of the URL can be changed or edited.
This new addition of the platform API endpoint will simply check the RSS feed URL authenticity and whether it works well or not before pushing it into the database.
import { scrapFromRSS } from 'utils';

const addNewPlatform = async(url) => {  
 const response = await scrapFromRSS(url);  
 const isValid = response.data.feeds.length > 0 ? true: false;  
 if(isValid){   
   // add this to the database collection of platforms  
  }else {   
  throw new Error('Invalid platform or RSS feed URL')  
}};
We have now automated the process to push the new platform easily to our database and aggregation collection.

Conclusion

This is just a 5-minute read to simply the way to develop a news aggregation platform.
  • Collect platform with RSS url whose data we want to fetch
  • Create an endpoint to fetch data using RSS scraper
  • Integrate the endpoint in the frontend and render all the feeds
  • Showcase the UI along with creating an endpoint to keep adding new platforms.
I’ve already created this platform but not sure whether I want to open it to the public because of the revenue stream and no business plan.
Once this is finalised I’ll push the platform will cool features along with cool UI design.
Do give it a try if it doesn’t work well it will become your good resume project to land a nice high-paying job.
See you in the next one
Shrey

4 steps plan of action

  • Collect the platforms you want to get onboard along with the link and RSS
  • Create an endpoint to scrap the data via the RSS URL, usually, one API call will work fine
  • Create a frontend interface to showcase all the fetched data from RSS urls of the onboarded platform
  • Integrate the endpoint API to the frontend interface and render/showcase all the platform's latest feeds or news in one place
Yes in these 4 steps, your aggregation or Google News clone platform is ready.

RSS feed Platforms

Collecting RSS feeds of the news platforms along with all the news platforms is an easy task.
You can use chatGPT for the same and get the extensive list of all platforms along with website, description, and RSS URL in one table or JSON format.
I did the same using the following prompt
Give me 30 news website in a table with the following as the column, RSS feed URL, 
description, website link and name
This will return a table but you can go with JSON format also.

Frontend Interface

Developing the frontend is not hard.
My preferred stack is the following
  • Next.js for framework
  • React.js language
  • Tailwind CSS for styling
  • Shadcn UI or Mantine.dev or Material UI or chakraUI for UI
  • Vercel for hosting
  • Github for version control
  • React framer for animation
  • Supabase or Firebase for database
We can use Firebase and Supabase in the above endpoint to store the latest feeds in the database to avoid making multiple calls to the endpoint.
Redis is a preferred choice for adding caching to the database.
CRON job can be used to schedule the refetching of the latest data from RSS feeds every hour.

Frontend API integration

The only part of the front end is to integrate the API.
Axios npm module will simply work well in this case.
const feeds = await axios.get(fetchDataFromRSS);
return feeds.data;

Subscribe

Our once a week newsletter on Programming, Jobs, AI, and Business