NZ Real Estate Data Scraper

Overview

Large-scale real estate data aggregation system scraping multiple NZ property websites including Trade Me, Ray White, and Harcourts. Distributed architecture with Celery task queue, Redis caching, and visual monitoring dashboard.

Objectives

• ▸ Extract property listings from major NZ real estate websites
• ▸ Normalize data across multiple sources
• ▸ Build comprehensive property database
• ▸ Distributed scraping with Celery workers
• ▸ Visual monitoring with Scrapydash dashboard

Architecture

Scrapy-based distributed scraping with Splash for JavaScript rendering. Celery workers process scraping tasks in parallel. Redis caching layer reduces database load. PostgreSQL stores normalized property data. Scrapydash provides visual monitoring.

Key Features

• ▸ Multi-website scraping (Trade Me, Ray White, Harcourts, OneRoof)
• ▸ JavaScript rendering with Splash for dynamic content
• ▸ Celery distributed task queue with Redis backend
• ▸ Scrapydash visual monitoring dashboard
• ▸ Data deduplication and normalization
• ▸ PostgreSQL database with proper indexing
• ▸ Periodic refresh and update operations

Challenges

• ▸ Maintaining scrapers against website structure changes
• ▸ JavaScript rendering complexity and captcha handling
• ▸ Legal and ethical considerations of scraping

Outcomes

• ▸ Comprehensive coverage of NZ real estate market
• ▸ Distributed architecture for scalability
• ▸ Visual monitoring and error tracking