🌐 Broadband-plan Querying Tool @ UCSB

Measuring broadband access, quality, and affordability at the street-address level — powering data-driven policy for a more connected America.

📡 Internet Inequity in the U.S.

The United States has long suffered from digital inequality stretching across multiple dimensions. Rural and tribal regions are far less likely than urban cities to have high-quality, high-speed Internet access. Several demographic and socioeconomic factors — including race, ethnicity, and income — are consistently linked to disparities in Internet availability and quality.

🎯 Countering Internet Inequity

To fully bridge the Internet access divide and best allocate available funding, policymakers need to understand the state of Internet availability, affordability, adoption, and quality. They require not only accurate distributions of Internet quality within a region, but also a clear picture of the contributors to the distribution’s tail (regions of poor quality).

Policymakers need data that enables a complete characterization of Internet access quality at the most refined geographical granularity — down to the street address level. Specifically, the data must answer:

Question
Where precisely does access exist (or not exist)?
What broadband plans are available in a given region?
Where do networks fail to meet established service goals?

🛠️ Broadband-plan Querying Tool (BQT)

BQT is a tool that queries the availability of broadband plans at any given street address, supporting. It simulates authentic user behavior and engages directly with ISP websites to extract ISP-provided speed and pricing data at street-address granularity. By curating a rich dataset spanning urban/rural locations and low/high income demographics, BQT unlocks deep insights into the affordability of ISP services — a dimension that has remained largely inaccessible due to a lack of appropriate datasets.

BQT Overview

Querying street addresses at scale introduces two major challenges:

🚧 Challenge 1 — IP Blocking

Conducting many queries from a single IP address can trigger blocks or rate limits imposed by ISPs, hindering scalability.

Solution: We partner with Bright Data, a leading proxy service provider. Through the Bright Initiative, they provide free access to data scraping tools for nonprofits and academic organizations — giving us a large pool of IP addresses to rotate across queries.

🚧 Challenge 2 — Slow Execution Time

Mimicking organic user workflow (rather than scraping ISP backends) takes tens of seconds per address. At scale, this becomes prohibitive.

Solution: We employ Docker containers and parallelize the querying process. For each ISP, we launch multiple Docker containers simultaneously, each handling its own subset of addresses — dramatically reducing total execution time.

BQT Docker Proxy Workflow

The figure above demonstrates the Docker + proxy workflow used to extract plan information from AT&T’s website.

📋 Emerging Policy Workloads

Broadband policy interventions introduce two policy workloads. First, assessing the efficacy of multi-billion-dollar policy interventions — such as the $44 billion Broadband Equity, Access, and Deployment (BEAD) program — requires accounting for a provider landscape of over 700 ISPs and interfaces. Furthermore, tracking state and local legislatures’ affordability interventions requires longitudinal monitoring of hundreds of ISPs, with minimal technical debt and computational cost.

This motivates two key requirements for any next-generation tool:

Requirement Description
🔌 Extensibility Must scale across 700+ ISPs and diverse web interfaces
🔁 Robustness Must sustain longitudinal monitoring without brittleness

🚀 A next-generation BQT

In order to support emerging policy workloads, BQT leverages agentic system design, decomposing complex ISP interaction processes into reusable, adaptable components that scale across various providers.

✨ Acknowledgments

This effort is supported by:

We are thankful to Bright Data for providing free access to their IP proxy service through the Bright Initiative. We extend our gratitude to CostQuest Associates and Zillow for sharing valuable street-address-level datasets.