Edd's Knowledge Base

Automatic Tag Generation and Clustering with LLMs

2 min read

In-memory semantic knowledge graph that converts personal documents into topic-aware discovery, visualizations, and reproducible experiment logs.

Quick Facts

  • Context: CS584 NLP Knowledge Graph Project
  • Tech Stack: Python 3.13+, OpenAI API, sentence-transformers (mpnet), KeyBERT, TextRank, NLTK, NetworkX, PyVis, NumPy, uv
  • Links: GitHub Repo | Project Report (PDF)

Overview and Problem

The project builds a lightweight knowledge graph from local documents to improve semantic search and topic discovery. It aims to evaluate different tag extraction strategies and similarity thresholds for optimal graph connectivity.

What I Built

  • Engineered an automated pipeline to extract semantic tags using LLM-backed generation (OpenAI API) with statistical fallbacks (KeyBERT, TextRank) and NLTK text preprocessing.
  • Implemented embeddings using sentence-transformers (mpnet models) to map tags into semantic space.
  • Designed a cosine matching algorithm with configurable k and threshold parameters to link documents to topic centroids.
  • Developed NetworkX graphs with interactive PyVis HTML exports and structured text dumps for visualization and debugging.

Architecture overview

flowchart LR
    DOC[Local documents] --> TAG[Tag extraction: LLM, KeyBERT, TextRank]
    TAG --> EMB[mpnet embeddings]
    EMB --> COS[Cosine matching: k and threshold]
    COS --> G[Knowledge graph]
    G --> VIZ[Interactive HTML export]
    G --> LOG[Experiment logs]

Key Results and Impact

  • Achieved best semantic coherence of 0.97 (LLM, k=3, threshold=0.7).
  • Reached strongest connectivity with an LCC ratio of 1.00 (TextRank, k=5 or k=7, threshold=0.3).
  • Handled dynamic topic counts ranging from 32 to 475 across thresholds, demonstrating a clear granularity tradeoff.

Core Learnings

  • Balanced midpoints yielded coherence around 0.77–0.81 with LCC 0.50–0.57 at k=3, threshold=0.5 across different extractors, showing the trade-offs between precision and connectivity.

Related: Projects MOC