Building a Stock Screener: Aggregating and transforming multi-country financial data in Python

Financial platforms need near zero-latency data streams to be useful at all. Aggregating stock indices, historical prices and corporate balance sheets across multiple countries introduces massive discrepancies in formatting, time zones, reporting cadence and structural integrity. A field that is populated for a US large-cap might be missing entirely for a small Saudi-listed company. A debt figure reported quarterly in one market turns up annually in another.

Our Sharia-compliant stock and crypto screening platform merges independent global data feeds into clean compliance verdicts in real time. It now serves over 100,000 registered users checking Indian, US, Saudi and UAE equities plus crypto assets against a single, defensible standard. Here is how the Python ETL pipeline behind it was engineered.

Ingestion: one handler per provider

Market data arrives from diverse providers, so the ingestion scripts are modular Python. Each provider connection is an independent feed handler that emits a standardized payload, and the same internal shape comes out whether the raw source was a REST API returning JSON, a bulk CSV export, or a paginated SOAP endpoint from an older exchange data vendor.

That modularity means a provider changing their API schema only forces an update inside that one handler, leaving the core compliance engine untouched. It has already paid for itself more than once. A provider serving one of the four regional markets restructured their response payload with no notice, and the fix was a one-file change to that provider's handler instead of a scramble across the whole pipeline.

Making the ratio maths fast enough

Sharia compliance comes down to a set of financial ratios: debt-to-assets, interest-income-to-revenue, illiquid-assets-to-total-assets. Each has to be calculated dynamically against a moving universe of thousands of equities across four markets. Run that over millions of records sequentially, in a plain Python loop, and a live screening dashboard will never stay current.

So we used Pandas and NumPy for vectorized array operations. Entire tables get processed concurrently in memory instead of looping through records individually, which brings compliance evaluation down to milliseconds per batch. Providers that return very large historical dumps are chunked into fixed-size batches before each chunk is vectorized. Peak memory on a single worker stays bounded that way, and nothing ever tries to load an entire market's history at once.

import pandas as pd
import numpy as np

def calculate_compliance(df_stocks):
    # Vectorized check: debt / total assets < 33%
    df_stocks['debt_ratio'] = df_stocks['total_debt'] / df_stocks['total_assets']
    df_stocks['is_compliant'] = np.where(df_stocks['debt_ratio'] < 0.33, True, False)
    
    return df_stocks[['symbol', 'debt_ratio', 'is_compliant']]

Currency and time zones

Global stock markets close at different hours and report financials in different local currencies. Our normalization pipeline converts all asset figures to USD using historical exchange rate API caches snapshotted at the time of publication. Not a live rate fetched at query time. A compliance verdict has to be reproducible, and a live-rate lookup would quietly change a historical verdict every time someone re-ran the calculation months later.

Every market record also carries its UTC timestamp. Historical charting queries then line up cleanly on one unified grid whatever the original market location, so a Riyadh close, a Mumbai close and a New York close all land on the same axis with no per-market special-casing downstream.

Data quality gates before a verdict ever ships

Not every incoming record deserves to reach the compliance engine. A provider outage that returns a stale cached response, a corporate action such as a stock split or a delisting that has not propagated through every field yet, a balance-sheet figure reported in the wrong currency by mistake: each of those produces a technically-valid-looking but wrong verdict if it goes straight through.

• Records whose reporting timestamp is older than that provider's expected refresh window get rejected outright. Silently reusing a stale figure is the worse outcome.
• A ratio that moves implausibly from the prior period is flagged, never dropped. A debt ratio that halves overnight deserves a human look.
• On a schedule, a sample of each provider's currency-converted figures is cross-checked against a second source. That is what catches a systematic conversion bug early, instead of after users notice it.

Recalculation is scheduled, not permanent

A compliance verdict computed last quarter can go stale the moment a company's balance sheet updates, so a stored verdict is never treated as final. The pipeline recalculates on a schedule. It re-evaluates every holding currently sitting on a user's watchlist or portfolio, and flags the moment a stock or coin crosses out of compliance instead of waiting for someone to check manually.

Caching and delivery

• Calculated stock status verdicts are cached in Redis, so client app queries answer instantly.
• Twice-daily cron pipelines refresh financial indicators and balance sheets.
• MongoDB indices on stock symbols and dates keep searches fast.

When processing global market data, do not compute sequentially. Use vectorized libraries and strong cache gates to keep response speeds fast.

How do you aggregate stock and crypto data from multiple countries in real time?

Each data provider is wrapped in its own modular ingestion script with a standardized output schema, so a vendor API change only touches one handler. Compliance ratios are then calculated with vectorized Pandas and NumPy operations instead of row-by-row loops, and the resulting verdicts are cached in Redis for instant delivery.