Multimodal Image Geolocation (Geo-HiPO)
Trained Vision-Language Models to predict where photos were taken using continuous physical distance signals, running experiments across multi-node H100 and V100 GPU clusters.
The problem
Standard image geolocation treats location prediction as multiple-choice classification. Under that setup, predicting a neighboring town scores as completely wrong, the same as guessing an opposite continent.
During a research appointment at the University of Central Florida Center for Research in Computer Vision (May 2025 - Aug 2025), I worked on Geo-HiPO. The goal was to train vision-language models to predict real-world coordinates using physical distance as the supervisory signal.
Method and preference optimization
We formulated geolocalization as Direct Preference Optimization (DPO) based on physical distance:
- The vision-language model generates candidate coordinate predictions for an input photo.
- Predictions are scored against ground truth using Haversine great-circle distance.
- The prediction closer to ground truth becomes the chosen sample and the farther prediction becomes the rejected sample for DPO.
- The training pipeline scales hierarchically, learning continents and countries first before fine-tuning on regional cities.
Research Poster Presentation
Published & Presented Research Poster
Cluster engineering and training stack
- Built multiprocessing data loaders in Python to clean and curate 30M+ images from MP-16 and IM2GPS3K, cutting over 100 hours of pipeline overhead.
- Filtered out invalid coordinates, ambiguous locations, and model hallucinations using GeoNames and OpenStreetMap datasets before training.
- Integrated vLLM to run batched candidate generation on Qwen2.5-VL and Kimi-VL checkpoints.
- Configured FlashAttention-2, PyTorch automatic mixed precision (AMP), and DeepSpeed to fit multimodal models across NVIDIA H100 and V100 clusters.
- Resolved GCC and CUDA version conflicts on cluster nodes and fixed token padding mismatches between image patch embeddings and text sequences.
Benchmark results
Evaluation on the IM2GPS3K benchmark showed clear gains over standard training:
- Qwen2.5-VL-7B with HiPO reached 35.87% accuracy within 25 km and 54.59% within 250 km, improving over the base model baselines of 34.13% and 52.22%.
- Standard supervised fine-tuning (SFT) degraded accuracy to 33.97%, demonstrating that distance-based preference ranking produces a stronger supervisory signal than token imitation.
Technologies used
- Vision-language models: Qwen2.5-VL (3B, 7B, 32B, 72B), Kimi-VL
- Deep learning frameworks: PyTorch, Hugging Face Transformers, Hugging Face Datasets, Hugging Face Accelerate, TRL, PEFT, LoRA
- Optimization and alignment: Direct Preference Optimization (DPO), SimPO, Supervised Fine-Tuning (SFT), Curriculum Learning
- Inference and acceleration: vLLM, DeepSpeed, FlashAttention-2, PyTorch AMP, bitsandbytes, FP8, INT8 Quantization
- HPC and cluster infrastructure: SLURM Workload Manager, NVIDIA H100 GPUs, NVIDIA V100 GPUs, Multi-Node Clusters, CUDA, GCC
- Computer vision and datasets: OpenCV, Pillow (PIL), Albumentations, GeoNames, OpenStreetMap (OSM), NumPy, pandas
- Systems and tooling: Linux (Ubuntu, RHEL), Docker, Conda, Git, Bash