ZeroMQ – High-Speed Subscribers

I’ve been messing around with Low latency and high frequency message processing and figured it might be worth sharing.  I created an example of the High-Speed Subscribers (Black Box Pattern) using ZeroMQ, “a high-performance asynchronous messaging library”.  According to its creator, you can pipe through 2 mil to 6 mil+ messages per second with one CPU core.

For some perspective, the New York Stock Exchange generates approx. 4 terabytes of data and in excess of 15 billion messages over the course of a single 6.5-hour trading day.   If you think that’s a lot, in Oct 2008 the NASDAQ TotalView -ITCH produced 999,814,794 messages in a single day.

High-Speed Subscriber Patterns in ZeroMQ

The Simple Black Box Pattern

Setup Environment & Run Example (Windows):

git clone https://github.com/ryansmccoy/zeromq-high-speed-subscribers
cd zeromq-high-speed-subscribers
conda create -n zeromq-high-speed-subscribers1 python=3.7 -y
activate zeromq-high-speed-subscribers1
pip install -r requirements.txt

Setup Environment & Run Example (Linux):

git clone https://github.com/ryansmccoy/zeromq-high-speed-subscribers
cd zeromq-high-speed-subscribers
conda create -n zeromq-high-speed-subscribers python=3.7 -y
pip install -r requirements.txt

To run, open seperate terminal windows, run:

Publisher

$   activate zeromq-high-speed-subscribers
$   python zeromq_high_speed_subscribers/00_publish_data_feed.py

Output:


Subscriber-to-Pusher

$   activate zeromq-high-speed-subscribers
$   python zeromq_high_speed_subscribers/01_subscriber_to_pusher_queue.py

Output:

Pull-to-Workers

$   activate zeromq-high-speed-subscribers
$   python zeromq_high_speed_subscribers/02_pull_to_workers.py

Output:

https://github.com/ryansmccoy/zeromq-high-speed-subscribers

Leave a Reply

Your email address will not be published. Required fields are marked *