Dashboard

Monitor your crawling activities and system status

Active Crawls
1
Products Collected
0
Scheduled Jobs
0
Errors Today
0

Recent Activity

  • New Herholdts.co.za crawler template created
    Specialized template for product extraction
    Just now

System Status

CPU Usage

35% - 4 cores @ 2.8GHz

Memory Usage

42% - 3.2GB / 7.6GB

Disk Usage

28% - 56GB / 200GB

Crawler Status

Crawler service is operational

New Web Crawl

Configure and launch a new web crawling job

Herholdts.co.za Product Crawler

https://

Crawl Parameters

Set to 0 for unlimited (not recommended)

Higher values reduce server load and detection risk

Maximum time to wait for each page to load

Advanced Options

Crawl Preview

Crawl Configuration Preview

Adjust settings to see how they affect your crawl

Target URL: herholdts.co.za
Data Points: 8 selected
Max Pages: 50
Request Delay: 3 seconds
Proxy: None

Sample Output Structure

{
  "products": [
    {
      "title": "Product Name",
      "price": "R1,299.00",
      "original_price": "R1,499.00",
      "description": "Detailed product description...",
      "images": [
        "https://herholdts.co.za/image1.jpg",
        "https://herholdts.co.za/image2.jpg"
      ],
      "sku": "PROD12345",
      "categories": ["Category 1", "Subcategory 1"],
      "specifications": {
        "Material": "Stainless Steel",
        "Dimensions": "30 x 20 x 15 cm",
        "Weight": "1.5 kg"
      },
      "availability": "In Stock",
      "url": "https://herholdts.co.za/product-url"
    }
  ],
  "metadata": {
    "crawl_date": "2023-06-20",
    "pages_crawled": 24,
    "products_found":