Introduction

Information Theory

Information Theory, application of mathematical principles to the problems of transmitting and storing information. Information theory stems originally from the work of American mathematician and electrical engineer Claude E. Shannon and, in particular, his classic paper “A Mathematical Theory of Communication,” published in 1948 in the Bell System Technical Journal. Shannon rewrote the paper and, with Warren Weaver, published it in book form as The Mathematical Theory of Communication in 1949. Shannon was interested in the problems of communicating information accurately from one place to another. He decided to approach communication from a mathematical point of view.

Information theory focuses on the problems inherent in sending and receiving messages and information. The theory is based on the idea that communication involves uncertain processes, both in the selection of the message to be transmitted and in the transmission of the message itself. Information theory provides a way to measure this uncertainty precisely.

In information theory, the actual meaning of the message is unimportant. Instead, the important qualities of communication are the amount of information that the message contains, the accuracy of the transmission, and the quality of the reception. All of these values are represented mathematically, so different messages and different communication systems can be compared, studied, and improved.

Information theory measures the amount of information in a message by using units called bits, short for binary digits, which use only the numbers 0 and 1 (see Number Systems). Information theory is useful because it provides a way to find the minimum number of bits required to communicate a given message. Information theory can also determine the maximum rate, in bits per second, at which a given communication channel can transmit reliable information.

Information theory is primarily a theoretical study. However, it has had a profound impact on the design of practical data communication and storage systems, such as telephones and computers. The theory can be applied to both the transmission and the storage of messages, because storage is nothing more than transmission in time. For example, both making a telephone call to a friend in another city and tape recording a message for a friend to play later in the day involve the same issues of sending and receiving messages. In information theory, no fundamental distinction is made between these two types of problems.