Building a Global Market Data Monitoring System with WebSocket Technology

December 22, 2024

table of contents

Construction and Revision of a Real-Time Stock Price Monitoring System

We ended up piecing together a system that displays real-time stock price information in a browser. Although it was created for the Japanese stock market, the same dreary approach can be applied to a variety of market data around the world. You can grudgingly check major indices, individual stock prices, price fluctuation rates, and sector data all at once, with a half-decent sector-based filtering feature and a bare-bones stock search function thrown in.

Global Applicability

Despite the hassle, the fundamental design of this system can be shoved into all sorts of market data scenarios, such as:

Stock market data from different countries
Commodity futures market prices
Foreign exchange rates
Cryptocurrency price data

The bottom line is that, as long as you can manage to secure a data source, the architecture here can be repurposed for pretty much any market. By combining it with the APIs or tools offered by local securities firms or data providers in your region, you could scrape together a similar system—assuming you really want to.

Real-time stock prices

Market Monitor

System’s Basic Mechanism

It’s essentially a hastily slapped-together setup that grabs market data from some source (like Excel or an API), funnels it through a web server, and dumps it in your browser. Whenever the data source decides to update, the browser grudgingly refreshes so you can witness the latest numbers—assuming that matters to you.

Key Features

Display of major indices (capable of begrudgingly supporting various representative indices)
Price and percentage-change tracking for individual stocks
Sector-based filtering (for those who bother)
Search function by stock name or code
A half-baked smartphone-friendly layout
Visual feedback when prices update (the bare minimum, really)

How to Use

Open the page in your browser
The top of the screen shows the index information
Below that, you’ll find a long, thrilling list of individual stocks
Use the sector filter if you’re desperate to narrow things down
Type into the search box if you actually know what you’re looking for

Caution

The displayed information is just for reference—don’t pin all your hopes on it
Update frequency depends entirely on whenever the data source gets around to it
Covered stocks only extend as far as the data source allows, so don’t expect miracles

Additional Note: About the Data Source

This system is, somewhat reluctantly, designed to plug into any market data source out there, such as:

Securities firm APIs
Market data vendor feeds
Info from the exchanges
Any other random source

Because the process of fetching, converting, and sending data is kept separate from the display side, it’s (theoretically) simple to swap or add new data sources. We’re currently stuck with Excel as our data source, which we only picked for testing during development. Luckily for you, switching to a more legitimate data source doesn’t require altering the display portion at all.

Program Structure

Overall, the system revolves around two Python programs—each doing its own questionable share of the workload.

Data Transmission Program

We slapped together a program that snatches market data from somewhere and shoves it into a WebSocket server. It’s “technically” built to handle the world’s major markets (NYSE, NASDAQ, BSE, NSE, TSE, etc.). By only transmitting updates when something changes, we barely keep the server from keeling over. Some half-hearted examples include:

U.S. Markets: Dow Jones Industrial Average (30), S&P 500, NASDAQ Composite, etc.
Indian Markets: SENSEX, NIFTY50, etc.
Asian Markets: Nikkei 225, Hang Seng Index, STI Index, etc.
Other: Individual stocks, ETFs, and so on

Data Reception and Distribution Program

This is a hastily assembled WebSocket server built with FastAPI that grudgingly spits out the received data to browsers in real time. It accommodates various global markets with awkward time differences and can technically deliver data to multiple clients at once. Feel free to connect via PC, smartphone, or tablet—if you really trust it.

Technical Features

A so-called “lightweight” Python-based implementation
Minimal modules (barely enough to function)
Real-time data updates via WebSocket
A flexible (if you can call it that) data structure for multi-market support
Some half-baked support for international time displays
Multiple currency support, for whatever that’s worth

In theory, as long as you manage to secure a data source, this program can display market data from anywhere on Earth. It can even juggle multiple market feeds at once—if you think that’s a good idea.

Development of a Futures & Derivatives Information Monitor

We halfheartedly put together a page that supposedly provides real-time info on derivatives (financial instruments). The system claims to accommodate futures trading data from various markets—though we’ll see how that goes.

System Features

Allegedly, this setup lets you keep an eye on the following, in “real time”:

Stock index futures (standard, mini, micro) prices and changes
Related index futures information
Major currency rates that might or might not be up to date
A lackluster visual display of order book data (buy/sell volume)
Transaction history (price changes over time, if you care)
A barely interactive price chart

Implementation Example: Japanese Market

For now, we’re grudgingly displaying Japanese futures market data as a test.

URL for “verification”: https://nk225.minokamo.tokyo
(Don’t bother checking during off-hours, holidays, or maintenance—because you’ll see nothing.)

Supposed Focus on Usability

Dark mode to reduce eye strain (we guess)
Responsive design so smartphones can suffer less
“Efficient” data updates for marginally smoother operation

Applicability

This system can theoretically be adapted to various derivatives markets, such as:

U.S. E-mini S&P 500 futures
Indian Nifty futures
European DAX futures
Other random commodity or derivative products

System Highlights

We only refresh the screen when data actually changes, so the server presumably uses fewer resources while still serving up “real-time” info. In other words, it’s an “efficient” design—when it actually works.

Evolution of a Market Data Storage System: A WebSocket Approach

We put together a system to store tick data (price, volume, and other time-series info) in real time, published in a rambling blog post and on GitHub. Whether you find it remotely helpful is another matter entirely.

Blog Post: https://minokamo.tokyo/2024/04/04/7075/
GitHub: https://github.com/superdoccimo/ayumi

This system supposedly supports data from a variety of markets around the globe—if you can be bothered.

Features of the Current System

Flexible Data Collection

Works with multiple data sources (exchange APIs, data vendors, trading tools, etc.)
So-called “efficient” data retrieval using Python and VBA
A half-baked design for multi-market compatibility

Advanced Data Management

Complete exclusion of duplicate data via hash checks—assuming it actually does what it says
Dual storage in Excel files and a database (MySQL/MariaDB) for reasons only we know
Low resource consumption for continuous, never-ending operation

Possible Markets

Stock Markets (NYSE, NASDAQ, BSE, NSE, TSE, and whatnot)
Commodity Futures
Forex
Cryptocurrency

With this (questionably) ingenious approach, you can collect and store all kinds of financial market data from around the world, not just Japan—handling each market’s quirks and time differences, or at least pretending to.

The “Innovation” of Market Data Collection via WebSocket

After some grueling technical analysis, we’ve concluded that WebSocket might be the “superior” approach for collecting market data—if only by process of elimination. Below are the main reasons (for whatever they’re worth):

Data Immediacy and Reliability

WebSocket approach: supposedly fetches market data in “real time,” down to the millisecond

Example: even during sudden price swings, it allegedly captures each tick
Could be stretched to handle high-frequency trades across global exchanges

Robust Error Handling

Real-time monitoring of connection status (so you know when it inevitably breaks)
An “intelligent” reconnection mechanism (we’ll see how smart it really is)
Stable operation even when markets go wild—if we’re lucky
Capable of limping along for 24-hour crypto markets, too

High Scalability

Simultaneous monitoring of global markets—assuming you want to juggle that mess
Manages data from multiple exchanges and markets all at once
“Advanced” features are supposedly easy to implement, like:
- Real-time market analysis—sure, if you trust the data
- A customizable alert system—for those who enjoy constant notifications
- Multi-currency display (because one currency wasn’t enough?)

System Resource Optimization

Efficient data stream management (in theory)
Minimal network overhead, if everything goes perfectly
Allegedly ideal for handling multi-market data at the same time

Future Outlook

We’re working on a “new version” that may or may not include:

Cross-market analysis (because one market is never chaotic enough)
Customizable data filters (for all your picky needs)
Multi-platform support (assuming it can even handle one properly)

Key Design Points

A data structure that’s supposed to accommodate global markets
Timestamp management that grudgingly accounts for time-zone differences
Support for multiple currencies—because you’ll probably need more than one
Compliance with international requirements (and we all know how thrilling that is)

Overview of the System Architecture

We’ve put together a fancy diagram (for show, mostly) to illustrate the basic setup and flow of this so-called global market data system.

sequenceDiagram
    participant Sources as Data Sources<br/>(Exchange/API/Tools)
    participant Collector as Data Collector
    participant Server as WebSocket Server
    participant Clients as Clients
    
    rect rgb(240, 240, 240)
        Note over Sources: Multi-source Support
        Sources->>Collector: Market Data
    end
    
    activate Collector
    Note over Collector: Data Standardization<br/>Deduplication
    Collector->>Server: TCP/WebSocket
    deactivate Collector
    
    activate Server
    rect rgb(240, 240, 240)
        Note over Server: Multi-client Distribution
        Server->>Clients: Real-time Data
        Clients-->>Server: Bidirectional Communication
    end
    deactivate Server

Data Flow Diagram

The lofty figure above purports to demonstrate how we collect and distribute market data from multiple sources. We get info from exchange APIs, trading tools, and some random feeds, then pretend to “standardize” it before pushing it out to clients.

System Architecture Diagram

This architecture supposedly enables the collection and distribution of global market data. Its main features include:

flowchart LR
    subgraph Data Sources
        Exchange[(Exchange API)]
        Tools[(Trading Tools)]
        Feed[(Market Feed)]
    end
    
    subgraph Backend Processing
        Collector[Data Collector]
        Server[WebSocket Server]
        Cache[(Cache System)]
    end
    
    subgraph Frontend Applications
        Web[Web Browser]
        Mobile[Mobile App]
        Trading[Trading Tools]
    end
    
    Exchange -->|Market Data| Collector
    Tools -->|Trading Data| Collector
    Feed -->|Information Feed| Collector
    
    Collector -->|Standardization| Server
    Server <-->|Cache| Cache
    
    Server -->|Real-time Data| Web
    Server -->|Real-time Data| Mobile
    Server -->|Real-time Data| Trading
    
    style Exchange fill:#f9f,stroke:#333,stroke-width:2px
    style Server fill:#bbf,stroke:#333,stroke-width:2px
    style Cache fill:#bfb,stroke:#333,stroke-width:2px
    style Web fill:#fbf,stroke:#333,stroke-width:2px

Multi-Source Support

Direct retrieval from exchange APIs (assuming they’re cooperative)
Data collection from trading tools (assuming they actually provide data)
Integration of various “market feeds”

Efficient Data Processing

Data “standardization” to make everything look consistent—at least on paper
A caching system for whatever marginal performance boost it offers
Removal of duplicate data, if the hashing actually works

Multi-Platform Distribution

Web browser compatibility (for all you folks still browsing)
Mobile app integration (because everyone wants push notifications, right?)
Trading tool integration (so you can funnel the chaos directly into your trades)

We claim this approach can deal with global market data, though that’s probably optimistic.