Web scraping over the Chrome DevTools Protocol
Scrapers for JavaScript-heavy sites with Playwright and Selenium, plus live Chrome sessions driven over a remote debugging port to capture network data.
- Status
- Production
- Period
- Mar 2025 – Feb 2026
- Role
- Data Scientist (AI & Automation), Nam Viet Media
- Code
- Proprietary
- Stack
- Playwright
- Selenium
- Chrome DevTools Protocol
- n8n
- Dynamic sites
- Playwright / Selenium
- Live Chrome via CDP
- Parsed data
- n8n, content pipeline, analytics DBs
Problem
The automation and analytics work needed data from dynamic, JavaScript-heavy sites, some of which require a logged-in session.
Approach
- Built scrapers with Playwright and Selenium.
- Controlled live Chrome sessions through the Chrome DevTools Protocol over a remote debugging port, to reuse logged-in sessions and capture network data.
- Parsed HTML and API responses.
Architecture
Scrapers feed their output into the n8n workflows, the LLM content pipeline, and the analytics databases.
Results
One collection layer serves three consumers: workflow automation, content generation, and analytics.