Syntax and Semantics

For many 'information' channels (e.g. the computer I'm using to type this, or a gas whose temperature is interesting) there is a huge disparity between the number of possible messages and the number of semantically useful messages that can be passed. There are many billions of states that my computer could be in that are entirely useless, and many many more billions of states that a cubic centimetre of air at 12 degrees Celsius can be in on a cool day in Aberdeen. Chaos and redundancy are intuitive barriers for many people coming to information theory for the first time: we instinctively think that 'information' means useful information - information that means something.

But we only get meaning through interpretation - the different states of the cubic centimetre of air are only interesting if we can distinguish them - by their consequences, or by decoding them in different ways.

This may be difficult with the gas example. We might think the computer example is easier - we can list the states of all the independent storage elements, and show where the differences lie.

But this doesn't meant that we can 'reverse engineer' all of these states and represent them as the intelligible output of a programme - even one with errors in in it.

Some states cannot arise as a result of programming errors (at any level) and can only result from hardware faults, but a single hardware fault might result in a large number of possible memory states, so that we only need to recognise them as members of a class in order to find the fault - we don't need to be able to identify and interpret each one separately.