Entropy and Relative Entropy from Information-Theoretic Principles

Publication Type:
Journal Article
IEEE Transactions on Information Theory, 2021, 67, (10), pp. 6313-6327
Issue Date:
Filename Description Size
Entropy_and_Relative_Entropy_From_Information-Theoretic_Principles.pdfPublished version729.38 kB
Adobe PDF
Full metadata record
We introduce an axiomatic approach to entropies and relative entropies that relies only on minimal information-theoretic axioms, namely monotonicity under mixing and data-processing as well as additivity for product distributions. We find that these axioms induce sufficient structure to establish continuity in the interior of the probability simplex and meaningful upper and lower bounds, e.g., we find that every relative entropy satisfying these axioms must lie between the Rényi divergences of order 0 and infty . We further show simple conditions for positive definiteness of such relative entropies and a characterisation in terms of a variant of relative trumping. Our main result is a one-to-one correspondence between entropies and relative entropies.
Please use this identifier to cite or link to this item: