Click any module to expand. The module is structured in three eras: Classical NLP (Modules 1–3), Transformer Era (Modules 4–6), and LLM-Powered NLP (Modules 7–9). Students understand WHY the field evolved and can choose the right approach for each problem — not every NLP task needs a 70B parameter model.
How computers process human language — tokenisation, cleaning, and representation
The NLP pipeline: raw text → preprocessing → representation → model → output. Python NLP ecosystem: NLTK (academic, comprehensive), spaCy (production-grade, fast), Hugging Face Transformers (modern, LLM-focused). When to use which: spaCy for production NER/POS, NLTK for learning, Hugging Face for everything Transformer-based. Setting up the environment: pip install spacy transformers datasets.
Word tokenisation: split on whitespace/punctuation. Problems: "New York" = 2 tokens, "don't" = 1 or 2? Subword tokenisation: BPE (Byte-Pair Encoding — used by GPT), WordPiece (used by BERT), SentencePiece (language-agnostic). Why subword matters: handles unseen words, reduces vocabulary size. Character-level for specific tasks. Tokenisation determines what the model "sees" — understanding it is fundamental.
Lowercasing, punctuation removal, stopword removal (when helpful, when harmful — removing "not" changes sentiment). Stemming (crude: running → run) vs lemmatisation (smart: better → good, ran → run). Regular expressions for pattern matching (emails, URLs, dates). Unicode handling, emoji processing. HTML/markdown stripping. Data-dependent: preprocessing for tweets ≠ preprocessing for legal documents.
Bag of Words: document = vector of word counts. Sparse, loses word order, but simple and effective baseline. TF-IDF: words that appear frequently in one document but rarely across documents get higher weight. N-grams: bigrams ("machine learning"), trigrams — capture local word order. CountVectorizer and TfidfVectorizer from scikit-learn. These representations STILL win for many classification tasks on small datasets.
Word2Vec: learn dense vectors where similar words have similar vectors. CBOW (predict word from context) vs Skip-gram (predict context from word). GloVe: global co-occurrence statistics → embeddings. FastText: subword embeddings — handles unseen words and morphological similarity. king - man + woman ≈ queen — the analogy test. Pretrained embeddings: download and use (don't train from scratch). The bridge from BoW to Transformers.
Classification, NER, POS tagging, and topic modelling — the practical NLP toolkit
Pipeline: TF-IDF → classifier (Logistic Regression, Naive Bayes, SVM, Random Forest). Sentiment analysis: positive/negative/neutral. Spam detection. Intent classification. Multi-label classification. Evaluation: accuracy, precision, recall, F1, confusion matrix. Cross-validation. Hyperparameter tuning. TF-IDF + Logistic Regression is STILL a strong baseline that beats many complex models on small datasets — always start here.
NER: identify and classify entities in text — Person, Organisation, Location, Date, Money. spaCy NER: nlp(text).ents — instant NER in one line. Custom NER: train spaCy on your own entity types (product names, medical terms, legal entities). BIO tagging scheme (Begin, Inside, Outside). Hugging Face NER models: token classification. Use cases: information extraction, document processing, knowledge graph construction.
POS tagging: assign grammatical role (noun, verb, adjective) to each word. spaCy: token.pos_, token.dep_. Dependency parsing: identify grammatical relationships (subject-verb-object). Visualisation: displacy.render(). Why POS/dependency still matters: rule-based information extraction, grammar checking, text analytics. When to use spaCy (fast, production) vs Stanza (research-grade).
LDA (Latent Dirichlet Allocation): discover latent topics in a document collection. Choosing number of topics: coherence score. Interpreting topics: top words per topic. BERTopic (modern): use sentence embeddings + clustering → topic discovery. BERTopic advantages: better topics, dynamic topics over time, hierarchical topics. Use cases: news categorisation, customer feedback analysis, research paper clustering.
Rule-based: VADER (Valence Aware Dictionary and sEntiment Reasoner) — works without training data. ML-based: TF-IDF + classifier. Transformer-based: Hugging Face sentiment pipeline. Aspect-based sentiment: "The food was great but service was terrible" → food: positive, service: negative. Multi-language sentiment. Building a complete sentiment analysis pipeline: data collection → preprocessing → model → evaluation → deployment.
Dense representations that capture meaning — the foundation of modern NLP applications
Sentence-BERT: encode entire sentences into fixed-size vectors where semantically similar sentences are close. SentenceTransformer('all-MiniLM-L6-v2'): the standard model (384 dimensions, fast). Cosine similarity for comparing embeddings. Use cases: semantic search, duplicate detection, clustering documents. Why sentence embeddings matter: TF-IDF misses "car" ≈ "automobile" — embeddings capture it.
Semantic search: find documents by MEANING, not keywords. Encode all documents → encode query → find nearest neighbours. FAISS (Facebook AI Similarity Search): efficient nearest-neighbour search on millions of vectors. Approximate nearest neighbours: HNSW algorithm. Beyond keyword search: "How to fix a leaking faucet" matches "plumbing repair guide." The technology behind modern search engines and recommendation systems.
Cluster documents by meaning: embed → K-Means/HDBSCAN → label clusters. Semantic deduplication: find near-duplicate documents across millions. Document similarity matrix for recommendations. Cross-lingual similarity: multilingual sentence embeddings match "Good morning" ≈ "Bonjour." Practical: customer support ticket clustering, research paper grouping, content deduplication.
The architecture that changed NLP — self-attention, encoders, decoders, and pre-training
Self-attention: Query, Key, Value matrices → attention scores (softmax(QK^T/√d)) → weighted sum of values. Multi-head attention: capture different relationship types in parallel. Positional encoding: inject sequence order (Transformers have no inherent position awareness). Feed-forward network. Layer normalisation. Residual connections. Encoder (bidirectional — BERT) vs decoder (autoregressive — GPT). The architecture paper: "Attention Is All You Need."
Pre-training: Masked Language Model (predict [MASK] tokens — 15% of input) + Next Sentence Prediction. BERT sees ALL tokens simultaneously (bidirectional). Fine-tuning: add task-specific head → train on small labelled data. BERT variants: DistilBERT (66% smaller, 97% performance), RoBERTa (better training recipe), ALBERT (parameter sharing), DeBERTa (disentangled attention — current state-of-the-art encoder). BPE tokenisation: WordPiece.
Pre-training: next-token prediction (predict the next word given all previous). Causal (masked) self-attention: can only attend to previous tokens. GPT scaling: GPT-2 (1.5B) → GPT-3 (175B) → GPT-4 (estimated 1.7T MoE). Emergent abilities: in-context learning, chain-of-thought reasoning appear at scale. BPE tokenisation. Temperature and top-k/top-p for generation diversity. The architecture behind ChatGPT, Claude, Gemini.
BPE (Byte-Pair Encoding): merge most frequent character pairs iteratively → build vocabulary. Used by GPT. WordPiece: similar but maximises likelihood of training data. Used by BERT. SentencePiece: language-agnostic, treats text as raw bytes. Tokeniser effects: "tokenisation" → ["token", "isation"] (BPE) vs ["token", "##isation"] (WordPiece). Why tokenisation matters: some languages need 4x more tokens than English → cost + quality implications.
The practical Hugging Face ecosystem — from pipeline() to Trainer API to custom models
pipeline("sentiment-analysis"), pipeline("ner"), pipeline("summarization"), pipeline("translation"), pipeline("zero-shot-classification"), pipeline("question-answering"), pipeline("text-generation"). Each pipeline: load model + tokeniser → preprocess → inference → postprocess — all automatic. The fastest way to get NLP results — and often sufficient for prototyping.
Dataset preparation: Hugging Face Datasets library (load_dataset). Tokenisation: AutoTokenizer with padding and truncation. Model: AutoModelForSequenceClassification. Trainer API: TrainingArguments + Trainer for managed training loop. Evaluation: accuracy, F1, confusion matrix. Training on Google Colab with GPU. Fine-tuning DistilBERT on IMDb sentiment: 3-minute training, 92%+ accuracy. The practical workflow every NLP engineer uses daily.
AutoModelForTokenClassification: per-token predictions. NER fine-tuning: B-PER, I-PER, B-ORG, I-ORG, O labels. Handling subword tokens: align labels with tokenised input. Custom entity types: train on your own annotated data. Evaluation: seqeval metrics (entity-level precision, recall, F1). Practical: fine-tune NER model for domain-specific entities (medical, legal, financial).
Text generation: model.generate() with greedy, beam search, top-k, nucleus (top-p) sampling. Temperature: low → conservative, high → creative. Summarisation: extractive (select important sentences) vs abstractive (generate new summary). Fine-tuning T5/BART for summarisation. ROUGE metrics (ROUGE-1, ROUGE-2, ROUGE-L) for evaluation. Machine translation with MarianMT/NLLB. The generation skills that connect to LLM applications.
Hugging Face Hub: 500K+ models, search by task/language/size. Model cards: understand model capabilities, limitations, biases. Datasets library: load any NLP benchmark (GLUE, SQuAD, CoNLL). evaluate library: compute metrics easily. Push to Hub: share your fine-tuned models. Model comparison: evaluate multiple models on same test set. The ecosystem that makes NLP research reproducible and accessible.
Production NLP tasks beyond classification — question answering, IE, and cross-lingual models
Extractive QA: find the answer span within a given context (SQuAD format). Model: AutoModelForQuestionAnswering + BERT/RoBERTa. Reading comprehension pipeline. Open-domain QA: retrieve relevant documents → extract answer (retriever-reader architecture — precursor to RAG). Evaluation: Exact Match (EM), F1 score. Building a QA system on custom documents.
Information extraction: structured data from unstructured text. Entity linking: connect NER results to knowledge bases (Wikipedia, Wikidata). Relation extraction: identify relationships between entities ("Apple" → CEO → "Tim Cook"). Triplet extraction: (subject, predicate, object). Knowledge graph construction from text. Use cases: legal document analysis, biomedical literature mining, news intelligence.
Multilingual models: mBERT (104 languages), XLM-RoBERTa (100 languages), IndicBERT (Indian languages). Zero-shot cross-lingual: train on English, test on Hindi — it works. Multilingual sentence embeddings: LaBSE, multilingual-e5. Indian language NLP: Hindi, Tamil, Bengali, Telugu — challenges and available resources. IndicNLP, AI4Bharat models. Why multilingual matters: serving India's 22+ language market.
Content moderation: detect toxic, offensive, hateful, or harmful content. Perspective API (Google), Hugging Face toxicity models. Multi-label: toxic AND insulting AND threatening — text can violate multiple categories. Bias in toxicity detection: models can flag African American English as "toxic" — fairness challenges. Data annotation for safety: guidelines, disagreement handling. The responsible NLP skill every social media and content company needs.
Using LLMs to solve NLP tasks WITHOUT fine-tuning — the paradigm shift of 2023–26
Zero-shot: "Classify this text as positive/negative: [text]" — no examples needed. Few-shot: provide 3–5 examples in the prompt → model learns the pattern. Chain-of-thought (CoT): "Think step by step" → better reasoning. System prompts for persona and constraints. Prompt templates for consistent results. When prompt engineering replaces fine-tuning (small datasets, rapid prototyping) vs when fine-tuning is better (large datasets, latency requirements, cost).
Calling LLM APIs from Python for NLP tasks: classification, extraction, summarisation, translation — all via prompts. Structured output: force JSON responses for programmatic use (OpenAI function calling, Anthropic tool use). Batch processing: classify 10,000 reviews using async API calls. Cost optimisation: choose model size by task complexity (GPT-4o-mini for classification, GPT-4o for complex reasoning). The API that turned NLP from a specialisation into a feature.
Fine-tuned BERT wins when: large labelled dataset, low latency required (<50ms), cost-sensitive (no per-token API fees), offline/edge deployment. LLM wins when: small/no labelled data, complex reasoning, rapid prototyping, multilingual (one model for all languages). Hybrid: LLM for labelling → fine-tune BERT on LLM-generated labels → deploy BERT. The decision framework every NLP engineer needs — "just use GPT" isn't always the right answer.
LLMs as annotators: classify, label, extract entities from unlabelled data at scale. LLM-as-judge: evaluate text quality, relevance, safety. Active learning: LLM labels → human reviews uncertain cases → model improves. Annotation guidelines: clear, consistent, calibrated. Tools: Label Studio, Argilla for annotation workflows. The modern annotation pipeline: LLM labels 90% → humans verify 10% → fine-tune production model.
Building production NLP applications powered by retrieval and LLMs
RAG architecture: chunk documents → embed → store in vector DB → user query → retrieve relevant chunks → LLM generates answer from context. Why RAG: LLMs hallucinate, RAG grounds them in YOUR data. Chunking strategies: fixed-size, recursive, semantic. Retrieval quality: recall@K, MRR. Generation quality: faithfulness (is the answer supported by context?), relevance. LangChain for building RAG pipelines. The #1 deployed NLP architecture in enterprise.
Hybrid search: combine keyword (BM25) + semantic (embedding) search. Re-ranking: cross-encoder reranks top-K retrieved chunks for precision. Multi-query RAG: rephrase user query in multiple ways → retrieve more diverse results. Parent-child chunking: retrieve small chunks, return surrounding context. Self-querying: LLM generates structured filters from natural language queries. The techniques that improve RAG from "sometimes works" to "production-ready."
Retrieval-based chatbots: match user query to predefined responses (FAQ bot). Generative chatbots: LLM generates responses in context. Memory: conversation history management (sliding window, summarisation). Guardrails: prevent the bot from going off-topic, generating harmful content, or hallucinating. Multi-turn dialogue management. Building a customer support chatbot: knowledge base → RAG → conversation → escalation to human.
Agents: LLM + tools (search, database query, code execution, API calls). ReAct pattern: Reasoning + Acting (think → act → observe → repeat). LangChain agents, LangGraph for complex workflows. NLP-specific agents: document analyser (upload PDF → extract entities → summarise → answer questions). Tool use: LLM decides which NLP pipeline to invoke based on user request. The agentic architecture that's the frontier of NLP applications.
LoRA fine-tuning, instruction tuning, and connecting text with speech
When prompt engineering isn't enough: domain-specific language, specialised output format, latency requirements. LoRA: add trainable low-rank matrices to attention layers — train 0.1–1% of parameters. QLoRA: 4-bit quantised base + LoRA — fine-tune 7B models on single GPU. Fine-tuning LLaMA/Mistral for domain-specific NLP: medical, legal, financial text. Evaluation: compare fine-tuned vs base model on task-specific benchmarks.
Instruction tuning: teach model to follow task instructions. Dataset format: {"instruction": "Extract all company names", "input": "...", "output": "..."}. Creating custom instruction datasets for NLP tasks. SFT (Supervised Fine-Tuning) with LoRA. DPO (Direct Preference Optimisation) for output quality. The process that transforms a base model into a task-following assistant — the bridge from pre-trained model to useful NLP tool.
Whisper (OpenAI): speech-to-text in 99 languages. Transcription pipeline: audio → Whisper → text → NLP analysis. Voice-powered applications: spoken customer feedback → transcribe → sentiment analysis → actionable insights. Speaker diarisation: who said what. Subtitling and captioning. Fine-tuning Whisper on domain-specific audio. The multimodal skill that connects audio and text NLP.
Metrics, production serving, and building a portfolio that gets NLP Engineer interviews
Classification: accuracy, precision, recall, F1, AUC-ROC. Generation: BLEU (n-gram overlap — translation), ROUGE (recall-oriented — summarisation), METEOR (semantic matching), BERTScore (embedding similarity). QA: Exact Match, F1 on token overlap. Human evaluation: when automated metrics aren't enough. LLM-as-judge: use GPT-4 to evaluate generation quality. Choosing the right metric for each NLP task.
FastAPI for model serving: /predict endpoint with Pydantic input validation. Gradio/Streamlit for interactive demos. Hugging Face Inference API: serve any Hub model via API (zero infrastructure). Docker for containerised deployment. Batching for throughput. Model caching for latency. Monitoring: input distribution drift, prediction confidence tracking. The production stack: develop (Colab) → serve (FastAPI) → demo (Gradio) → host (HF Spaces/Railway).
Portfolio structure: 4 projects showing progression (classical NLP → fine-tuned Transformer → RAG application → LLM agent). Each: GitHub repo + Hugging Face model card + deployed demo. Kaggle notebooks with NLP competitions. Blog posts explaining NLP concepts and project decisions. The portfolio that demonstrates breadth (multiple NLP tasks) AND depth (production-quality code with evaluation).
TF-IDF baseline → fine-tuned DistilBERT → zero-shot LLM comparison. English + Hindi support. Complete evaluation with confusion matrices. Deployed via Gradio.
Fine-tune NER model on medical/legal/financial entities. Data annotation with Label Studio. spaCy + Hugging Face comparison. Entity linking to knowledge base.
Multi-document RAG: chunk → embed → ChromaDB → hybrid retrieval → LLM answer with citations. Advanced: re-ranking, multi-query. Evaluation: faithfulness + relevance scoring.
Customer support chatbot: RAG for knowledge, conversation memory, topic guardrails, escalation to human. LLM-powered with safety filters.
Trainers build NLP pipelines live — from text preprocessing to fine-tuning BERT to deploying RAG applications. Real datasets, real problems, real code.
Each week: solve an NLP task on a new dataset. Sentiment analysis, NER, summarisation, QA — rotate through real-world NLP problems.
Trainers review: choice of approach (classical vs Transformer vs LLM), evaluation methodology, error analysis, deployment readiness.
Students deploy NLP models and applications as interactive Gradio demos. Live URLs for the portfolio. "Try my NLP model" wins interviews.
ChatGPT, Claude, Gemini, Copilot — every generative AI product processes and generates TEXT. NLP is the foundation of the GenAI revolution. Students who understand NLP (tokenisation, embeddings, attention, fine-tuning, RAG) are positioned for the highest-growth AI career path.
NLP Engineer, AI Engineer, Applied Scientist — roles at Google, Amazon, Flipkart, and startups offer ₹15–40L+ packages. NLP specialists who can fine-tune models, build RAG systems, and deploy production applications are among the most sought-after technical profiles in 2025–26.
Legal (contract analysis), healthcare (clinical NER), finance (sentiment from news), e-commerce (review analysis, chatbots), media (content moderation), customer support (automated response). NLP isn't limited to tech companies — every industry that processes text needs NLP engineers.
Sentiment analyser, NER system, RAG Q&A, chatbot with guardrails — 4 deployed projects that demonstrate progression from classical to LLM-powered NLP. Hugging Face model cards, Gradio demos, and clean evaluation. The NLP portfolio that gets interviews at AI companies..
E-305, Industrial Area, Sahibzada Ajit Singh Nagar, Punjab 160055, IN
