LogoGeorgia Tech | Internet Intelligence Lab
R-N-X InetIntel Website Folder/
BGPFinder

BGPFinder

BGPFinder: BGPStream-compatible BGP Data Indexer & Broker

BGPFinder is a high-performance, database-backed indexer and broker for BGP MRT data archives. BGPFinder was designed to act as a drop-in replacement for the BGPStream Broker and fully implements version 2 of the BGPStream API.
With BGPFinder, you can query both historical and real-time BGP data using the libbgpstream or pybgpstream libraries. Existing BGPStream clients that use these libraries will be able to switch to using a BGPFinder broker with zero code modifications.
BGPFinder is open source, and we encourage users to host their own BGPFinder instance wherever possible for improved local performance, security, and throughput. The Internet Intelligence Lab also hosts a public instance at https://bgpfinder.inetintel.cc.gatech.edu which is available for general purpose use.
The source code for BGPFinder is hosted on Github.

Architecture

BGPFinder consists of three components:
  • Scraper: A periodic daemon that crawls the web archives of University of Oregon RouteViews and RIPE RIS to discover new BGP dumps in near real-time. In the future, we also hope to add support for other public BGP data sources.
  • Database: A PostgreSQL database that stores the metadata for all of the scraped RIB and UPDATE files, including the URL corresponding to where each file was found.
  • API Service: A lightweight Go-based HTTP server that implements the BGPStream Broker v2 API, and responds to BGPStream client requests by querying the database and returning the set of file URLs which match the request parameters.

Public Broker Instance

The Internet Intelligence Lab at the Georgia Institute of Technology runs a public, high-capacity BGPFinder broker instance for the community: https://bgpfinder.inetintel.cc.gatech.edu
You are welcome to query this public broker for both historical analysis and real-time BGP data queries.

How to Use the Public Broker in Python

If you are using pybgpstream, simply set the broker parameter:
import pybgpstream

# Initialize the stream using the public BGPFinder broker
stream = pybgpstream.BGPStream(
    broker="https://bgpfinder.inetintel.cc.gatech.edu",
    filter="collector route-views.sg"
)

# Start reading BGP records
for record in stream.records():
    print(record.project, record.collector, record.time)
In software that is based on libbgpstream (such as bgpreader), you can change the broker url using the -o parameter when providing your input options, for example:
bgpreader -w 1744979200,1745065600 -t ribs -c rrc03 -d broker -o url=https://bgpfinder.inetintel.cc.gatech.edu

Self-Hosting BGPFinder

For local development, high-throughput queries, or to simply avoid reliance on an external broker, you can also self-host BGPFinder using the Docker compose configuration included in our Github repository.
A BGPFinder broker service (including scraper) requires relatively few compute resources. 8 CPU cores and 16GB of RAM should be more than sufficient to run all 3 components (database, scraper, and API service). At the time of writing, the entire URL database for both Routeviews and RIPE RIS consumes just over 8GB of disk space.

Prerequisites

  • Docker and Docker Compose installed on your host.
  • Outbound network access to RouteViews and RIS data archives (note, we recommend hosting a local mirror of these archives if possible but bear in mind that this will require a LOT of disk space).

Self-Hosting Steps

1. Configure the Environment
Copy the example environment file and customize it for your needs:
cp example.env .env
Open .env and set: * POSTGRES_PASSWORD: A secure password for your database. * API_PORT: The public port for the API service (e.g., 8080). * DB_DATA_PATH: The host path for persistent database storage (e.g., ./pgdata).
2. Launch the Stack
Start all components (API, scraper, database) in detached mode:
docker-compose up -d --build
On the first startup, PostgreSQL automatically runs migrations to prepare the database schema, and the services automatically align their health checks before handling traffic.
3. Verify Deployment
Verify that the containers are running:
docker-compose ps
You can query your local broker’s metadata to ensure it’s functional:
curl "http://localhost:8080/meta/projects?human=1"
When you start BGPFinder for the first time, it will take a fair while for the scraper to fully populate the database with all of the historical Routeviews and RIS data.
Note that these deployment instructions are for a simple instance where all components run on a single host with no redundancy. Extending this to support a more robust deployment that is suitable for hosting public instances is left an an exercise for the reader, but feel free to reach out to the Internet Intelligence Lab if you would like advice on this topic.

Development and Maintenance

BGPFinder was originally designed and implemented by Alistair King.
The following people have made additional contributions that turned Alistair’s original idea into the software that you see here today:
  • Shane Alcock
  • Zachary Bischof
  • Nahush Kumta
  • Xie Qiu

Maintenance & Support

BGPFinder is currently maintained by the Internet Intelligence Lab at the Georgia Institute of Technology.