Entropy and Relative Entropy from Information-Theoretic Principles

Gour, G; Tomamichel, M

Entropy and Relative Entropy from Information-Theoretic Principles

Gour, G Tomamichel, M

Permalink

Publisher:: IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
Publication Type:: Journal Article
Citation:: IEEE Transactions on Information Theory, 2021, 67, (10), pp. 6313-6327
Issue Date:: 2021-10-01

Closed Access

	Filename	Description	Size
	Entropy_and_Relative_Entropy_From_Information-Theoretic_Principles.pdf	Published version	729.38 kB	Adobe PDF	View/Open

Copyright Clearance Process

Recently Added
In Progress
Closed Access

This item is closed access and not available.

Full metadata record

Field	Value	Language
dc.contributor.author	Gour, G
dc.contributor.author	Tomamichel, M https://orcid.org/0000-0001-5410-3329
dc.date.accessioned	2022-06-03T03:48:21Z
dc.date.available	2022-06-03T03:48:21Z
dc.date.issued	2021-10-01
dc.identifier.citation	IEEE Transactions on Information Theory, 2021, 67, (10), pp. 6313-6327
dc.identifier.issn	0018-9448
dc.identifier.issn	1557-9654
dc.identifier.uri	http://hdl.handle.net/10453/157898
dc.description.abstract	We introduce an axiomatic approach to entropies and relative entropies that relies only on minimal information-theoretic axioms, namely monotonicity under mixing and data-processing as well as additivity for product distributions. We find that these axioms induce sufficient structure to establish continuity in the interior of the probability simplex and meaningful upper and lower bounds, e.g., we find that every relative entropy satisfying these axioms must lie between the Rényi divergences of order 0 and infty . We further show simple conditions for positive definiteness of such relative entropies and a characterisation in terms of a variant of relative trumping. Our main result is a one-to-one correspondence between entropies and relative entropies.
dc.language	English
dc.publisher	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
dc.relation.ispartof	IEEE Transactions on Information Theory
dc.relation.isbasedon	10.1109/TIT.2021.3078337
dc.rights	info:eu-repo/semantics/closedAccess
dc.subject	0801 Artificial Intelligence and Image Processing, 0906 Electrical and Electronic Engineering, 1005 Communications Technologies
dc.subject.classification	Networking & Telecommunications
dc.title	Entropy and Relative Entropy from Information-Theoretic Principles
dc.type	Journal Article
utslib.citation.volume	67
utslib.for	0801 Artificial Intelligence and Image Processing
utslib.for	0906 Electrical and Electronic Engineering
utslib.for	1005 Communications Technologies
pubs.organisational-group	/University of Technology Sydney
pubs.organisational-group	/University of Technology Sydney/Faculty of Engineering and Information Technology
utslib.copyright.status	closed_access	*
dc.date.updated	2022-06-03T03:48:19Z
pubs.issue	10
pubs.publication-status	Published
pubs.volume	67
utslib.citation.issue	10

Abstract:

We introduce an axiomatic approach to entropies and relative entropies that relies only on minimal information-theoretic axioms, namely monotonicity under mixing and data-processing as well as additivity for product distributions. We find that these axioms induce sufficient structure to establish continuity in the interior of the probability simplex and meaningful upper and lower bounds, e.g., we find that every relative entropy satisfying these axioms must lie between the Rényi divergences of order 0 and infty . We further show simple conditions for positive definiteness of such relative entropies and a characterisation in terms of a variant of relative trumping. Our main result is a one-to-one correspondence between entropies and relative entropies.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/157898