Until recently, we are pretty much familiar with physicals quantities and their properties, such as mass, energy, forces etc. But we are not very familiar with a term, called information, which is as important as other physical quantities although we use it in our everyday life without knowing or thinking about it very much. But giving attention to this abstract quantity and formulating theories and ideas has led us to a better and new exciting era- the information age.
What is information? Why is it so important to us? Why is it different from other physical quantities? These are the questions which will probably occur to the readers when they first hear about “information”. Information, generally speaking, is an abstract quantity which defines the possibility (not probability) of states. For example, the possible states of flipping a coin are two: either “Head” or “Tail”. So, we can define the basic unit for information is bit – which can represent two possible states. Maybe, the reader can raise a question if it is possible to define a unit which can represent one possible state. But after thinking carefully, we will see that there is no point in defining a unit which can represent only one possible state as it will nullify the ideas of possibility.
In fact, possibility means we only know the possible states, but we do not know in advance what state will actually happen. Also, all possible states are independent of each other as in flipping coin example. The possible states can be either “Head” or “Tail”, but cannot be both.
Once we understand the basic unit for information, we can go for next step – to define infinitely many possible states. The idea is simple:
2^n , where n is the integer: 1, 2, 3, 4 …
We can define as many possible sates as we want to. We just need to raise a power of 2. So, 1 bit of information represents 2 possible states. 2 bit of information represents 4 possible states and so on.
But it is very important to understand the difference between the unit of information and the information itself. It is the same as the difference between the unit of length and the length itself. Information is an abstract quantity to represent possibility and bit is the basic unit of information to measure information. For example, 4 bit of information has more possible states than 1 bit of information.
Once we understand the concept of possibility, we can link it to the probability. Since, as we know already, more bits can represent more possible states, the chance of happening for each state becomes less. In other words, more bits mean less probability. Therefore, we can say that information is inversely varied with probability.
Information∝ 1/Probability
Let's think about the above formula to verify if it makes sense or not. In our every day life, we usually consider strange incidents or events which are rare to happen as news or new information. In other worlds, events with low probability convey a lot of information to us. If we hear a new-born baby has 5 legs, we will probably think that it is news to us.
In fact, information not only represent the possibility of states, but also describes the pattern or periodicity of possibility. Since the possible states of 1 bit of information can be either 1 or 0 (on or off), a series of 1s and 0s can represent repetitive pattern of higher level information. For example, in order to represent North, East, West and South, 2 bit of information is required i.e., 00,01,10,11. Again, North, East, West, South can represent another level of information, such as North-East, South-West. So, int this way, higher level of information can be encoded with lower level of information.
In fact, we can see such kind of information encoding and decoding in our every day life. The most common information encoding and decoding is our language. Our language is composed of the smallest unbreakable unit - letters. Then, words are formed from letters, and then sentences are formed from words, and paragraphs and so forth. By using various patterns, our language can convey almost all possible events or incidents happening around us.
Also, the distribution of the patterns of information also defines the amount of information we need to represent. In fact, various encoding schemes have been used to represent letters. Here, please do not confuse information and information encoding. Information encoding uses the unit of information to represent information - the possible states or events. In the days of telegraph, Morse Code was widely used to convey information. Morse Code employs variable length encoding scheme i.e., it uses variable length bit pattern to encode the information. However, with the introduction to ASCII Code, fixed encoding has become wide-spread.
The key difference fixed encoding and variable encoding is that variable encoding will allow us to compress the information.
What is information? Why is it so important to us? Why is it different from other physical quantities? These are the questions which will probably occur to the readers when they first hear about “information”. Information, generally speaking, is an abstract quantity which defines the possibility (not probability) of states. For example, the possible states of flipping a coin are two: either “Head” or “Tail”. So, we can define the basic unit for information is bit – which can represent two possible states. Maybe, the reader can raise a question if it is possible to define a unit which can represent one possible state. But after thinking carefully, we will see that there is no point in defining a unit which can represent only one possible state as it will nullify the ideas of possibility.
In fact, possibility means we only know the possible states, but we do not know in advance what state will actually happen. Also, all possible states are independent of each other as in flipping coin example. The possible states can be either “Head” or “Tail”, but cannot be both.
Once we understand the basic unit for information, we can go for next step – to define infinitely many possible states. The idea is simple:
2^n , where n is the integer: 1, 2, 3, 4 …
We can define as many possible sates as we want to. We just need to raise a power of 2. So, 1 bit of information represents 2 possible states. 2 bit of information represents 4 possible states and so on.
But it is very important to understand the difference between the unit of information and the information itself. It is the same as the difference between the unit of length and the length itself. Information is an abstract quantity to represent possibility and bit is the basic unit of information to measure information. For example, 4 bit of information has more possible states than 1 bit of information.
Once we understand the concept of possibility, we can link it to the probability. Since, as we know already, more bits can represent more possible states, the chance of happening for each state becomes less. In other words, more bits mean less probability. Therefore, we can say that information is inversely varied with probability.
Information∝ 1/Probability
Let's think about the above formula to verify if it makes sense or not. In our every day life, we usually consider strange incidents or events which are rare to happen as news or new information. In other worlds, events with low probability convey a lot of information to us. If we hear a new-born baby has 5 legs, we will probably think that it is news to us.
In fact, information not only represent the possibility of states, but also describes the pattern or periodicity of possibility. Since the possible states of 1 bit of information can be either 1 or 0 (on or off), a series of 1s and 0s can represent repetitive pattern of higher level information. For example, in order to represent North, East, West and South, 2 bit of information is required i.e., 00,01,10,11. Again, North, East, West, South can represent another level of information, such as North-East, South-West. So, int this way, higher level of information can be encoded with lower level of information.
In fact, we can see such kind of information encoding and decoding in our every day life. The most common information encoding and decoding is our language. Our language is composed of the smallest unbreakable unit - letters. Then, words are formed from letters, and then sentences are formed from words, and paragraphs and so forth. By using various patterns, our language can convey almost all possible events or incidents happening around us.
Also, the distribution of the patterns of information also defines the amount of information we need to represent. In fact, various encoding schemes have been used to represent letters. Here, please do not confuse information and information encoding. Information encoding uses the unit of information to represent information - the possible states or events. In the days of telegraph, Morse Code was widely used to convey information. Morse Code employs variable length encoding scheme i.e., it uses variable length bit pattern to encode the information. However, with the introduction to ASCII Code, fixed encoding has become wide-spread.
The key difference fixed encoding and variable encoding is that variable encoding will allow us to compress the information.