1
Home
Bits That Matter
1
Home
2
Introduction
The Language of Uncertainty
3
Chapter 1: What Is Information?
4
Chapter 2: Entropy — Measuring the Unknowable
5
Chapter 3: Bits, Nats, and Bans
Compression: Entropy in Action
6
Chapter 4: Why Data Compresses (and When It Won’t)
7
Chapter 5: Codes and Coding
8
Chapter 6: Arithmetic Coding and Beyond
9
Chapter 7: Kolmogorov Complexity — The Uncomputable Ideal
Communication: Sending Information Reliably
10
Chapter 8: The Channel Model
11
Chapter 9: Error Detection and Correction
12
Chapter 10: Channel Capacity in Practice
Inference: Information as a Thinking Tool
13
Chapter 11: Relative Entropy and KL Divergence
14
Chapter 12: Mutual Information
15
Chapter 13: The Minimum Description Length Principle
Information Theory in the Wild
16
Chapter 14: Entropy in Cryptography
17
Chapter 15: Information Theory in Machine Learning
18
Chapter 16: Databases, Indexes, and Selectivity
19
Chapter 17: Designing Information-Dense Systems
Appendices
A
Appendix A: Mathematical Notation Reference
B
Appendix B: Python Toolkit
C
Appendix C: Annotated Further Reading
D
Appendix D: Worked Solutions to Chapter Exercises
Table of contents
Bits That Matter
Information Theory for Programmers
Author
Vijay Mathew
Published
April 3, 2026
2
Introduction