Aug 2024 - May 2025
Custom LLM and RAG Architecture
Trained Transformer language models from scratch in PyTorch, built a pgvector retrieval-augmented generation pipeline on verified documentation, and presented attention mechanism research to university leadership.
The goal
To study language model internals, I implemented autoregressive Transformer models from scratch in PyTorch. The project focused on training compact models on local GPUs and connecting them to a retrieval-augmented generation (RAG) pipeline using pgvector in PostgreSQL to ground responses in technical documentation and reference manuals.
Transformer architecture from scratch
- Wrote the Transformer architecture directly in PyTorch, implementing multi-head self-attention, scaled dot-product calculations, and rotary positional encodings.
- Built Byte-Pair Encoding (BPE) tokenizers and custom token-packing data loaders to eliminate padding waste and keep GPU utilization high during training.
- Evaluated parameter-efficient fine-tuning using LoRA and QLoRA adapters with 4-bit and 8-bit quantization (bitsandbytes) on local hardware.
Semantic search and document grounding
- Built a document ingestion pipeline that parses, chunks, and embeds technical manuals into dense vector representations.
- Stored embeddings using pgvector in PostgreSQL and implemented cosine distance queries to retrieve relevant reference passages.
- Added a semantic reranking step to prioritize top passages, ensuring answers cite verified source material.
Faculty presentation and research
- Analyzed attention weight distributions across network layers to evaluate how self-attention heads capture syntactic and semantic dependencies.
- Authored a formal research poster and delivered technical demonstrations to university faculty, department chairs, and the university president.
Educational AI Enablement & LLM Architecture Research Poster
Presented to LSSU leadership and faculty explaining Transformer architecture and retrieval pipelines
Research Poster View
Technologies used
- Deep learning and NLP: PyTorch, Hugging Face Transformers, Hugging Face Datasets, Hugging Face Accelerate, Multi-Head Self-Attention, Positional Encodings
- Tokenization and data loaders: Byte-Pair Encoding (BPE), Token Packing, Custom Data Loaders, NumPy
- Parameter-efficient fine-tuning: PEFT, LoRA, QLoRA, bitsandbytes (4-Bit and 8-Bit Quantization)
- Retrieval and vector search: PostgreSQL, pgvector, Sentence Embeddings, Cosine Similarity, Semantic Reranking, Document Chunking, RAG
- Languages and systems: Python, CUDA, C++, Bash, Linux, Conda, Git