When AI Audited My Thesis: The Retrospective Clicked in a Missing Puzzle Piece

As DeepWiki accurately summarized, I built a raw batch ETL and analytics pipeline. I ingested massive 1 TB XML 7z data dumps of StackOverflow posts, performing heavy data wrangling using JSoup and regex to strip HTML and extract clean text. From there, I enriched the data with sentiment and topic labels, managed complex nested HashMaps to aggregate counts and merge duplicate topics (e.g., merging topics 10+24), and wrote custom data quality validation logic to compare input/output record counts to guarantee zero data loss.

DeepWiki exposed the exact enterprise bottlenecks: the pipeline was modular but entirely manually orchestrated. File paths were hardcoded—completely killing portability—and downstream popularity metrics were calculated manually in SPSS rather than programmatically. It was a functional ETL pipeline, but it lacked data observability and the automated scheduling required of a reliable, enterprise-grade data product.

Cloud-native batch processing and the ELT (Extract, Load, Transform) paradigm. Today, hardcoded paths and manual SPSS steps are entirely eliminated. Raw data is programmatically ingested into a Google Cloud Storage (GCS) Data Lake. Instead of writing brittle Java HashMaps to aggregate topics, transformations are handled using declarative, partitioned SQL models inside BigQuery, treating the entire workflow as a Directed Acyclic Graph (DAG) automatically scheduled and monitored via Bruin. Most importantly, I now approach this architecture knowing that Data Platform Engineers don't build pipelines in a vacuum—we build governed, automated platforms designed specifically to unblock internal customers (Data Scientists, BI Analysts) and accelerate the company's time-to-insight.

I built a functional, research-grade batch pipeline that tightly coupled compute and execution chained together multiple external tools (JSoup, Senti4SD, Mallet LDA). I pushed a single 32GB RAM Windows machine to its absolute physical limits to ensure the data wasn't being silently dropped mid-flight while local hardware didn't crash from memory exhaustion during massive workloads.

DeepWiki’s audit exposed the exact infrastructure risks: the system was a localized monolith with zero environment parity. Hardcoded absolute file paths made the pipeline non-portable, and external dependencies weren't containerized, creating massive reproducibility risks—if my laptop died, the project died. The orchestration was entirely manual—there was no scheduler, no automated retries, and no structured logging. The execution lacked observability (no structured logging or telemetry). Furthermore, the pipeline was not idempotent; rerunning a failed stage meant risking duplicate outputs, and having separate manual pipelines for different datasets (Big Data vs. Concurrency) duplicated platform technical debt.

Immutable infrastructure and Infrastructure as Code (IaC) with Containerization and declarative orchestration. Today, the execution environment is never a single laptop—it is containerized via Docker and compute resources are dynamically provisioned on Kubernetes (GKE) using OpenTofu or Terraform or CloudFormation. Hardcoded paths are replaced by environment variables. Drawing from my background in hardware-level testing, I now design for fault tolerance. Manual execution is replaced by a modern workflow engine (Bruin) that enforces idempotency, handles retries, and treats data pipelines as directed acyclic graphs (DAGs) state is managed safely, and the system can gracefully recover from node failures without manual intervention.

I materialized a clean, topic-aware sentiment model from raw XML. By establishing a clear data grain, I built a reliable fact table joining post IDs to sentiment labels generated by Senti4SD, and integrated topic assignments from an external LDA model. From there, I built a functional metric layer: handling duplicate topic merges, applying basic dimensional modeling, and aggregating sentiment counts to produce a final dataset ready for Kendall Tau rank correlations in SPSS.

DeepWiki’s audit accurately pointed out the lack of analytics governance and data lineage. While intermediate CSVs were used for debugging, the pipeline completely lacked dbt-like versioning, isolated staging environments, and automated data testing. Crucially, the final enrichment step—joining the downstream popularity and difficulty metrics—was done manually inside SPSS rather than being programmatically integrated as core business logic within the pipeline.

Decoupled transformations and automated metric joins. What I used to call "data cleaning" is now governed as a true Semantic Layer, unblocks Data Analysts from doing repetitive SQL debugging. Instead of relying on manual SPSS interventions, I write modular, version-controlled, and heavily tested SQL models directly in a cloud data warehouse like BigQuery. Complex joins and metric aggregations are handled declaratively via Bruin, ensuring that downstream Business Intelligence (BI) tools and dashboards consume a fully automated, governed source of truth.

I built a heavy batch NLP batch inference pipeline to process roughly 400,000 posts. It handled deep text normalization using JSoup and orchestrated two external models: Senti4SD for sentiment classification and Mallet LDA for topic modeling. Because of hardware limits, I had to manually split the files, run the batch inference, and use complex HashMap lookups with position-based keys (e.g., matching row t0 to t0) to stitch the sentiment predictions back to the original post IDs.

DeepWiki exposed the exact fragility of this research-grade approach. First, position-based matching is incredibly risky in production—if a single row silently drops during inference, the entire downstream alignment breaks. Second, the system completely lacked model lifecycle management. There was no Model Registry, no artifact tracking for the Senti4SD or LDA models, and zero NLP-specific observability to monitor model degradation or data drift (like tracking shifting sentiment confidence scores).

ID-preserving, containerized ML pipelines. Today, I would never rely on position-based matching; feature engineering and transformations must be ID-preserving by design. Machine learning models are no longer treated as loose external scripts; they are containerized, pinned to specific versions, and managed as tracked model artifacts allow Data Scientists to deploy models faster without worrying about infrastructure. Instead of dumping data into fragile CSVs, intermediate outputs are saved in structured, columnar formats like Parquet, with the entire inference lifecycle orchestrated dynamically to prevent manual file chunking.

To accommodate the memory limits of the Senti4SD tool, I had to hardcode file splits at 10,000 records each. Processing over 150,000 posts meant running multiple Java stages, executing redundant I/O passes over the same text, and generating a massive proliferation of intermediate CSV files for questions, answers, and sentiment mappings.

In a university setting, the only cost is the electric bill, but DeepWiki’s audit highlighted the massive infrastructure and opportunity cost of this architecture. Manual orchestration meant idle resources waiting for human intervention, killing compute utilization. Hardcoded splits led to inefficient small-file operations, and redundant I/O passes amplified storage bloat and data egress overhead. In an enterprise cloud environment, this lack of storage lifecycle management would cause cloud bills to hemorrhage money.

Rightsizing compute and resource efficiency. True FinOps isn't just about minimizing cloud bills; it is about reducing time-to-insight so the business can make faster, data-driven decisions without scaling infrastructure costs linearly. Today, I replace fixed file splits with dynamic batching matched to instance memory. Intermediate CSVs are replaced by optimized columnar formats or managed BigQuery storage to slash I/O costs. Furthermore, by automating scheduling via Bruin, I eliminate manual wait times and can leverage preemptible spot instances via OpenTofu to process heavy batch workloads for pennies on the dollar, proving out a highly efficient chargeback model.