Web Scraping

7 articles in this category

PythonWeb ScrapingPlaywright

Codexity Part 4: Web Scraping, Proxies, and Anti-Bot Warfare

Fetch and extract content from 15 web pages concurrently. Handle JavaScript rendering with Playwright, dodge anti-bot systems, rotate proxies, and strip HTML down to clean text using readability-lxml and BeautifulSoup.

Aug 4, 2026

AI NewsWeb ScrapingFinTech

Building a Low-Cost Pipeline for U.S. Congress Trading Data

Engineer Fatih İlhan develops Apify actors to scrape congressional trades from government sources, reducing data costs to just $0.72 per day.

Apr 30, 2026

AI NewsWeb ScrapingAutomation

Scraping SAM.gov and USASpending for Federal Contracts via Python

Automate federal contract tracking using Python to merge SAM.gov and USASpending data, capturing $700 billion in annual opportunities without mandatory API keys.

Apr 18, 2026

AI NewsWeb ScrapingSoftware Development

2026 Guide to Anti-Bot Detection: Lessons from 34 Production Scrapers

Analysis of 34 production scrapers serving 300+ users, highlighting strategies to bypass Cloudflare and DataDome using Crawlee and residential proxies.

Mar 18, 2026

AI NewsWeb ScrapingDevOps

Overcoming IP Bans in Web Scraping Without Budget by Building a Resilient API Layer

Building a reverse proxy API for dynamic IP management can help overcome IP bans in web scraping, with a success rate of up to 90%.

Feb 3, 2026

AI NewsBot DevelopmentWeb Scraping

Building an LLM-powered Facebook Marketplace Bot

A Facebook Marketplace bot was built using LLMs for filtering, achieving perfect results with gpt-4o-mini.

Dec 26, 2025

AI NewsWeb ScrapingAutomation

Unlocking Stable Data Collection: The Dual Strategy of AI Browsers and CAPTCHA Solvers

Achieve 99% success rates in web scraping by combining AI Browsers with CAPTCHA solving services.

Nov 25, 2025