Building a Stock Screener: Aggregating and transforming multi-country financial data in Python

Financial platforms require near zero-latency data streams to be useful. Aggregating stock indices, historical prices, and corporate sheets across multiple countries introduces massive discrepancies in formatting, time zones, and structural integrity.

Our Sharia-compliant stock and crypto screening platform solves this by merging independent global data feeds into clean compliance verdicts in real time. This is how we engineered the Python ETL data pipeline.

1. Multi-Source Ingestion Engine

To fetch market data feeds from diverse providers, we designed modular ingestion scripts in Python. Each provider connection is treated as an independent feed handler that outputs a standardized payload.

This modularity ensures that if one provider changes their API schema, only that single handler needs updating, leaving the core compliance engine untouched.

2. High-Performance Calculations

Evaluating Sharia compliance requires calculating financial ratios (debt-to-assets, interest-income-to-revenue) dynamically. Performing these mathematical calculations on millions of records sequentially is too slow.

We utilized Pandas and NumPy in Python to perform vectorized array operations. Instead of looping through records individually, entire tables are processed concurrently in memory, reducing compliance evaluation time to milliseconds.

import pandas as pd
import numpy as np

def calculate_compliance(df_stocks):
    # Vectorized check: debt / total assets < 33%
    df_stocks['debt_ratio'] = df_stocks['total_debt'] / df_stocks['total_assets']
    df_stocks['is_compliant'] = np.where(df_stocks['debt_ratio'] < 0.33, True, False)
    
    return df_stocks[['symbol', 'debt_ratio', 'is_compliant']]

3. Normalizing currency and time zones

Global stock markets close at different times and report financials in different local currencies. Our normalization pipeline converts all asset figures to USD using historical exchange rate API caches at the time of publication.

We also tag every market record with its UTC timestamp, ensuring historical charting queries line up cleanly on a single unified grid regardless of the original market location.

4. Caching and Delivery

• Cache calculated stock status verdicts in Redis for instant client app query responses.
• Run twice-daily cron pipelines to update financial indicators and balance sheets.
• Optimize MongoDB indices on stock symbols and dates to support fast searches.

When processing global market data, do not compute sequentially. Use vectorized libraries and strong cache gates to keep response speeds fast.

How do you aggregate stock and crypto data from multiple countries in real time?

Each data provider is wrapped in its own modular ingestion script with a standardized output schema, so a vendor API change only touches one handler. Compliance ratios are then calculated with vectorized Pandas/NumPy operations instead of row-by-row loops, and the resulting verdicts are cached in Redis for instant delivery.