Course Outline

Instructor: Prof. Andrew Thangaraj

Contents:

• Information measures
  • Introduction - iid repetitions, high probability sets
  • Entropy, Relative entropy, Mutual information
  • Chain rules, Properties and inequalities
• Source Coding
  • Data compression problem
  • Discrete memoryless sources
  • Source codes - rate, length
  • Prefix condition and Kraft inequality
  • Huffman codes
  • Source coding theorem
• Channel Coding
  • Data transmission problem
  • Discrete memoryless channels
  • Channel codes - rate, length, error probability
  • Random codes
  • Robust typicality
  • Joint and conditional typicality
  • Channel coding theorem - optimality and capacity
  • Differential entropy, Gaussian channels and capacity
• Statistics
  • Learning from data
  • Binary hypothesis testing, Neyman-Pearson paradigm
  • Chernoff-Stein lemma
  • Parameter estimation, Minimax optimality
  • Le Cam lower bound, Prior lower bound

Evaluation & Books

• Assessment pattern:
  • 2 Quizzes (25% each) and a Endsem (50%), as per the institute schedule.
  • Optional Project
• Textbooks: Elements of Information Theory by Thomas and Cover.