Real-Time AQI Prediction
Real-Time Air Quality Monitoring & Prediction Pipeline
An end-to-end real-time air quality prediction pipeline combining streaming data, distributed processing, machine learning, and interactive visualization.
From Live Data to Prediction
The end-to-end pipeline continuously fetches live weather and pollutant data, processes it via PySpark streaming, predicts the Air Quality Index using a Random Forest model, and updates a dashboard in real-time.
Pipeline Architecture
01 / Data Ingestion
Data is gathered from the OpenWeatherMap API every 15 seconds. The custom `producer.py` script orchestrates the collection of both weather metrics and air pollution statistics, writing them as JSON objects into a designated streaming directory.
while True:
weather_data = fetch_weather()
pollution_data = fetch_pollution()
record = {
"timestamp": time.time(),
"weather": weather_data,
"pollution": pollution_data
}
write_to_stream(record)
time.sleep(15)02 / Stream Processing with Spark
Apache Spark Structured Streaming reads the incoming JSON files, flattens nested structures, validates schemas, and handles null values before appending the clean records to a Parquet storage layer for ML training.
03 / AQI Prediction
- Temperature
- Humidity
- Wind Speed
- Pressure
- Clouds
- PM2.5
- PM10
- CO
- NO₂
- O₃
Random Forest Regressor
| Algorithm | Random Forest Regressor |
| Estimators | 100 trees |
| Max Depth | 7 |
| Target | AQI |
04 / Real-Time Monitoring
Real-Time AQI Predictions
| Time | AQI | PM2.5 | Temp |
|---|---|---|---|
| 10:45:30 | 78 | 18.3 | 28.5 |
| 10:45:15 | 77 | 18.1 | 28.4 |
| 10:45:00 | 75 | 17.9 | 28.4 |
Behind the Pipeline
Fetches data via API
Spark stream ingestion
Storage layer
Model training
Serialized model
Live inference & dash
Engineering Challenges
Continuous Data Ingestion
Handling rate limits and connection drops while fetching from OpenWeatherMap every 15s continuously.
Structured Streaming Data
Transforming complex nested JSON arrays into flat schemas suitable for model inference using PySpark.
Real-Time Visualization
Updating the Streamlit dashboard asynchronously as new predictions arrive without causing page reloads.
What This Project Demonstrates
- End-to-end data pipeline construction
- Micro-batch streaming with Spark
- Real-time API integration
- Handling nested JSON schemas
- Parquet file optimization
- Machine learning regression
- Feature engineering from streams
- Model serialization and inference
- Streamlit real-time dashboarding
- Resilient error handling
Technology Stack
Explore the Real-Time AQI Pipeline Codebase
View the PySpark streaming data pipeline, Random Forest prediction model, and Streamlit dashboard code on GitHub.