Skip to main content

​
Component overview

┌──────────────────────────────────────────────────────────────┐
│  ProxyServer  (raw asyncio TCP)                              │
│                                                              │
│  _dispatch → RequestHandler (ABC)                           │
│    ├── ConnectTunnelHandler   CONNECT method → blind relay   │
│    ├── ForwardingHandler      /https/host/path → full retry  │
│    └── HttpProxyHandler       http://... → full retry        │
└──────────────────────────────┬───────────────────────────────┘
                               │ submit(PendingRequest)
          ┌────────────────────▼────────────────────┐
          │    TargetManager  (one per target)       │
          │    asyncio queue + dispatcher            │
          │    aiohttp outbound requests + retries   │
          └────────────────────┬────────────────────┘
                               │ acquire / record_success / record_failure
          ┌────────────────────▼────────────────────┐
          │    IPPool  (one per target)              │
          │    quarantine policy, cooldown sweeps    │
          └────────────────────┬────────────────────┘
                               │ push / pop / counters
          ┌────────────────────▼────────────────────┐
          │    IPPoolBackend                         │
          │    Memory | Redis                        │
          └─────────────────────────────────────────┘

​
Key components

ProxyServer — raw asyncio TCP server. Accepts connections, peeks at the first bytes to determine the interaction mode, and dispatches to the appropriate RequestHandler. RequestHandler (ABC) — each interaction mode is a self-contained class. Adding a new mode requires only a new subclass registered in handlers.py — ProxyServer needs no changes. ConnectTunnelHandler — handles CONNECT method requests. Establishes a TCP tunnel through an external proxy IP and relays bytes blindly. Cannot retry mid-flight failures. ForwardingHandler — handles X-Proxy-Hopper-Target requests. Reconstructs the full target URL and makes the HTTPS request via aiohttp, enabling retries on failure. HttpProxyHandler — handles plain http:// proxy requests. Forwards via aiohttp with full retry support. TargetManager — one per target. Maintains an asyncio queue of pending requests and dispatches them using IPPool.acquire(). Runs the aiohttp outbound requests and handles retry logic. IPPool — one per target. All quarantine and cooldown policy lives here. Never touches the backend directly — delegates storage to IPPoolBackend. IPPoolBackend — pure storage interface. Two implementations: MemoryBackend (asyncio queues + dicts) and RedisBackend (BLPOP queues + sorted sets). IPProber — background task that periodically probes each IP through the actual proxy to verify reachability. Updates ip_reachable metrics and can trigger quarantine on repeated probe failures.

​
Design principles

  • Mode isolation — each interaction mode is fully self-contained. The server has no mode-specific logic.
  • Policy isolation — quarantine and cooldown policy lives entirely in IPPool. The backend is pure storage with no policy.
  • Backend abstraction — IPPoolBackend is a clean interface. Swapping backends requires no changes above the pool layer.
  • No shared mutable state between targets — each target has its own TargetManager and IPPool instance. There is no global lock.