An Introduction to Information Theory: Symbols, Signals and Noise
- Downloads：6934
- Type：Epub+TxT+PDF+Mobi
- Create Date：2021-05-05 11:58:26
- Update Date：2025-09-06
- Status：finish
- Author：John Robinson Pierce
- ISBN：0486240614
- Environment：PC/Android/iPhone/iPad/Kindle
- Download
Summary
Behind the familiar surfaces of the telephone， radio， and television lies a sophisticated and intriguing body of knowledge known as information theory。 This is the theory that has permitted the rapid development of all sorts of communication， from color television to the clear transmission of photographs from the vicinity of Jupiter。
To give a solid introduction to this burgeoning field， J。 R。 Pierce has revised his well-received 1961 study of information theory for a second edition。 Beginning with the origins of the field， Dr。 Pierce follows the brilliant formulations of Claude Shannon and describes such aspects of the subject as encoding and binary digits， entropy， language and meaning， efficient encoding， and the noisy channel。 He then goes beyond the strict confines of the topic to explore the ways in which information theory relates to physics， cybernetics， psychology， and art。
Mathematical formulas are introduced at the appropriate points for the benefit of serious students。 A glossary of terms and an appendix on mathematical notation are provided to help the less mathematically sophisticated。
J。 R。 Pierce worked for many years at the Bell Telephone Laboratories， where he became Director of Research in Communications Principles。 His Introduction to Information Theory continues to be the most impressive nontechnical account available and a fascinating introduction to the subject for lay readers。
To give a solid introduction to this burgeoning field， J。 R。 Pierce has revised his well-received 1961 study of information theory for a second edition。 Beginning with the origins of the field， Dr。 Pierce follows the brilliant formulations of Claude Shannon and describes such aspects of the subject as encoding and binary digits， entropy， language and meaning， efficient encoding， and the noisy channel。 He then goes beyond the strict confines of the topic to explore the ways in which information theory relates to physics， cybernetics， psychology， and art。
Mathematical formulas are introduced at the appropriate points for the benefit of serious students。 A glossary of terms and an appendix on mathematical notation are provided to help the less mathematically sophisticated。
J。 R。 Pierce worked for many years at the Bell Telephone Laboratories， where he became Director of Research in Communications Principles。 His Introduction to Information Theory continues to be the most impressive nontechnical account available and a fascinating introduction to the subject for lay readers。
Reviews
Joseph Schoolland
，
I know more now about information theory than before I read this， but a lot of this book was over my head。
John Allard
，
Hard to rate， the parts about the actual mathematics of information theory are rather good （but not great given the standards of modern mathematical texts） but the rest of the book is quite dated。 I would look elsewhere for an introduction to the very interesting field。
Jarred Finney
，
Read this to understand acoustic signaling and used that knowledge to build an amp。 If you are interested in Claude Shannon this is very interesting。 Fortunes Formula is also a great read if you want to learn more about these pioneers。
Karen
，
It seems the main purpose of this book is to provide evidence that Information / Communication Theory should be taken seriously as a real science。 Perhaps because I'm reading this so many years after its writing， all I can think is， "how could it not？" The author provides a warning in the beginning that the book contains a lot of math， but because I don't have a strong math background I tend to let those parts of books like these flow over and through and pick up what I can when it gets back to It seems the main purpose of this book is to provide evidence that Information / Communication Theory should be taken seriously as a real science。 Perhaps because I'm reading this so many years after its writing， all I can think is， "how could it not？" The author provides a warning in the beginning that the book contains a lot of math， but because I don't have a strong math background I tend to let those parts of books like these flow over and through and pick up what I can when it gets back to the theory portions。 Nonetheless， as an audiobook， it's especially challenging to listen to recitations of long strings and formulas。I particularly appreciate the historical reference the book provides and learning about the process involved in calculating how to move data from a sender to a receiver through a wide variety of mediums。 。。。more
Mark Bates
，
Difficulty to follow。 Does not follow clear path。
Caleb
，
A good introduction to information theory。 I hadn't read anything about it before so I thought this was pretty good。 A good introduction to information theory。 I hadn't read anything about it before so I thought this was pretty good。 。。。more
Ahmad
，
Very informative introduction to the subject， I really liked the material until the author started connecting information theory to other fields， such as: psychology， cybernetics and art。 I would have appreciated a deeper dive into coding theory and telecommunication systems。
Jack
，
Interesting addition to my libraryThis book is as much a philosophical narrative on information theory as it is a technical treatise。 All in all， it’s a nice addition to my information theory library and it helps with my seemingly never ending effort to truly understand this stuff。 I have a strong bent toward practical applications and I’m not sure this book helped much in that regard。
Landis
，
Pierce offers here both a philosophical and mathematical account of first principles of information theory at the same time as he sketches an historical overview of the field which veers as much towards hagiography of Shannon as Pierce will allow himself to veer， given his stark plain-spokenness。 Pierce doesn't offer a new approach to information theory， but he is masterful at presenting a concise synopsis of and introduction to the problems and quandaries of the field， foremost among which seem Pierce offers here both a philosophical and mathematical account of first principles of information theory at the same time as he sketches an historical overview of the field which veers as much towards hagiography of Shannon as Pierce will allow himself to veer， given his stark plain-spokenness。 Pierce doesn't offer a new approach to information theory， but he is masterful at presenting a concise synopsis of and introduction to the problems and quandaries of the field， foremost among which seem to be the two eternal conundrums of the information theorist， namely， how to pack as much information into as few bits as possible and， second， how to use good old science and ingenuity to outwit the eternal foe of information theory: noise。 Pierce is fun to read because you can easily imagine a whole tradition of American language-users from the Puritans to Hemingway reading and endorsing what seem to be Pierce's practical potions for avoiding wastefulness with words and casting out confounded Satanic noise。 So， where others may come to information theory and find a dry， dull science， in Pierce I found the standard of that uniquely Protestant and uniquely American insistence on economy of language being raised in another arena beyond literature。 And it made me proud。 。。。more
Gy
，
It's always fun to delve into something written in the '80s。 It was a nice attempt from Mr。 Pierce to give an intro into communication theory。 That's not an easy task since communication theory is a mathematical as well。 Let's put it as， encoding was an average， with a little be too many binary bits。。。although， let us admit， that from point of view of mathematical readiness， the channel was perhaps too noisy。 But， the message came through。Generally speaking， there was some intentional degree of It's always fun to delve into something written in the '80s。 It was a nice attempt from Mr。 Pierce to give an intro into communication theory。 That's not an easy task since communication theory is a mathematical as well。 Let's put it as， encoding was an average， with a little be too many binary bits。。。although， let us admit， that from point of view of mathematical readiness， the channel was perhaps too noisy。 But， the message came through。Generally speaking， there was some intentional degree of obfuscation that did some part appear pretty Cimmerian， and prolonged reading and comprehending a bit longer， yet the read is enjoyable。 。。。more
Alexander Miles
，
Several conversations with co-workers convinced me that I should try to get at least a passing familiarity with information theory， if only for my own benefit。 I chose this particular book， honestly because it was easily available， with decent reviews， and comparatively inexpensive next to textbooks on the subject。 The author goes to solid lengths to avoid dropping too much math on the reader， and when it is unavoidable， does a great job getting the interpretation across。 While a small chunk of Several conversations with co-workers convinced me that I should try to get at least a passing familiarity with information theory， if only for my own benefit。 I chose this particular book， honestly because it was easily available， with decent reviews， and comparatively inexpensive next to textbooks on the subject。 The author goes to solid lengths to avoid dropping too much math on the reader， and when it is unavoidable， does a great job getting the interpretation across。 While a small chunk of the book was material I was already familiar with （namely Fourier transforms and their properties）， the rest was almost entirely new to me。 Over all， I can say I'm quite happy with the book。There are a handful of chapters， specifically 10-13， where the author departs from the nuts-and-bolts of information theory to cover more of the applications。 Some of these are quite concrete and instructive， like the chapter on information theory in physics。 Others， well。。 not so much。 They were entertaining to read， the author waxes poetic on the ideas of cybernetics （with a meaning pulled from the 1960s that the reader will likely have to acclimate to） and art， and not without some skill， just don't expect more coverage of technical topics there。I've been meaning to double-back on the first handful of chapters while taking notes， as I'd like to get the specifics to really sink in， and follow it up with one of Shannon's original papers， which I'm told I should now have enough context to really appreciate。 All-in-all， Pierce's introduction was a clear and entertaining entry point to the field。 I can recommend this book to anyone who has a passing interest in the idea of information theory， but doesn't want to dive head-first into the academic literature。 。。。more
Peter Rollins
，
A surprisingly accessible yet fairly in depth introduction to Claude Shannon’s Information Theory， getting both into the nitty gritty details， while also giving a broad overview of it’s usage within a broad range of fields。。。
Ryan Frantz
，
This was a bit of a tough read for me。 I appreciated the author's conversational tone， but some of the writing was difficult to parse。 And when it came to the maths， I am woefully unprepared to ingest it。 However， there were a few morsels I was able to extract， such as information theory's variation on the idea of entropy， that will lead me to explore other books on this topic that I hope are more approachable for me。 I may even brush up on my maths so I stand a chance of interpreting future tex This was a bit of a tough read for me。 I appreciated the author's conversational tone， but some of the writing was difficult to parse。 And when it came to the maths， I am woefully unprepared to ingest it。 However， there were a few morsels I was able to extract， such as information theory's variation on the idea of entropy， that will lead me to explore other books on this topic that I hope are more approachable for me。 I may even brush up on my maths so I stand a chance of interpreting future texts' use of it to explain these theories。 。。。more
CRAWFORD COLLINS
，
Great introduction to the field。 Wide ranging and no more math than necessary。
Alex Zakharov
，
Terrific and short introduction to the subject via an incredibly precise and clear writing。 As advertised it is intended for a general audience， but it is not a dumbed-down pop-sci book - you'll need a fair amount of grit to get through it。 Perhaps the best takeaway from the book is clearing of the nonsense and confusion that permeates 95% of "entropy" discussions in print and digital media。 In one of the early chapters Pierce summarizes and contrasts the usage of the term in physics and communi Terrific and short introduction to the subject via an incredibly precise and clear writing。 As advertised it is intended for a general audience， but it is not a dumbed-down pop-sci book - you'll need a fair amount of grit to get through it。 Perhaps the best takeaway from the book is clearing of the nonsense and confusion that permeates 95% of "entropy" discussions in print and digital media。 In one of the early chapters Pierce summarizes and contrasts the usage of the term in physics and communication theory。 In physics entropy is associated with irreversibility of certain processes， and if entropy increases available free energy decreases。 In statistical mechanics an increase of entropy is interpreted as increase in disorder and a decrease in our knowledge （think of gas filling up the second chamber of a 2-chamber container once the separation is removed）。 Communication theory is all about message sources， message receivers， and amount of information in the messages。 In that context entropy is the amount of uncertainty associated with the message that a receiver can get。 The higher the information content of the message， the higher the uncertainty as to which message a producer will send， and the larger the entropy。 In the main body of the book Pierce takes us from Fourier analysis and linear circuits to ergodic sources （thank you Kolmogoroff and Wiener）， efficient message encoding （e。g。 Huffman codes， hyperquantization， vocoders etc）， sampling theorem and of course Claude Shannon， noisy channels and channel capacity （via brilliant hyperspaces-inspired geometrical proof）。 At the tail end Pierce serves us a delicious treat by bringing information theory and physics together。 He notes that while information theory is fundamentally a math theory， in the end communication must happen in our physical reality with imposes unavoidable limitations。 He proceeds to construct a single-molecule variant of Maxwell demon where we can seemingly escape 2nd law of thermodynamics， if we know just one bit of information （which of course has certain communication-theory entropy）。 But Johnson noise （electromagnetic noise emitted by any non-zero temperature body） incurs an unavoidable energy cost on transmitting that message。 In Pierce's construction we need just 1 bit of information， and then we can reduce the statistical-mechanics entropy of the system and get free energy， but that free energy is exactly equal to the energy we would need to transmit that 1 bit over a fundamentally thermally-noisy channel！ Tasty！Note， the book is not nearly as dry as I make it sound， and it has plenty of excursions into natural language and meaning， art， “cybernetics”， and psychology。 I love it how Pierce sprinkles opinionated pearls here and there， for example contrasting skepticism of general public （a masked confusion） vs skepticism of scientists （clarity and avoidance of nonsense）。 You’ll also get a bonus discussion of James Joyce and Zipf’s law plus Mozart and rule-based random waltzes generation。 Finally you probably wouldn’t be too surprised to find out that stochastic art is often preferable to mediocrity。 Hoo-ah， what’s not to love！ 。。。more
Betawolf
，
_Symbols， Signals and Noise_ lies somewhere in the aether between textbooks and popular science。 It is certainly more easy to read than many examples of the former， with careful explanations in plain language of the phenomena being discussed， and intuitive explanations of any important equations or discoveries， yet it is also certainly a deeper treatment than a swathe of modern books which present only gaudy tidbits of information and some patronising examples。 Pierce stresses that his book woul _Symbols， Signals and Noise_ lies somewhere in the aether between textbooks and popular science。 It is certainly more easy to read than many examples of the former， with careful explanations in plain language of the phenomena being discussed， and intuitive explanations of any important equations or discoveries， yet it is also certainly a deeper treatment than a swathe of modern books which present only gaudy tidbits of information and some patronising examples。 Pierce stresses that his book would be empty without maths， and aims to fill in the gaps in laymen's understanding rather than skim over them -- even if he does omit proofs in many cases。 Most notable about the book is its readable style。 Rarely does treatment of a scientific topic have this narrative pull which keeps you reading， or such a sense of an author's voice。 It is a pleasure to learn from this character， whose intuitive explanations， scientific modesty and occasional spot of mild humour all work to educate you about a deep and broad topic， both in its core components as established by Shannon and in its applications to fields as diverse as linguistics， psychology and physics。 The book is not without flaws， but these are for the most part textual。 The terms 'information theory' and 'communication theory' are both used， seemingly interchangeably throughout the book， with no real explanation of why this is so。 There are a number of typographical errors in unfortunately critical areas -- numbers which are mistyped， symbols which are exchanged in the middle of important explanations。 I entertained for a while that the book itself was demonstrating a form of error correction -- the redundancy in the text being sufficient to allow the detection and correction of errors。 In his chapter on cybernetics， Pierce discusses for a while computing and predictive programs， and his treatment here has dated quite amusingly， not that it is any fault of his that the history of computation has changed since the book's publication， or that problems he saw unsolved have been tackled with great energy。 Certainly a great read for anyone with an interest in the topic， and I daresay a valuable guide to understanding for an undergraduate in communications or computing， or a professional moving from another field。 。。。more
'Special' Ed Harris
，
Good introduction to information theory。I'm quite certain I'd have been more benefited by this book if I had taken the time to also do the math， but it was a worthy read nonetheless。 Good introduction to information theory。I'm quite certain I'd have been more benefited by this book if I had taken the time to also do the math， but it was a worthy read nonetheless。 。。。more
Robert Pres
，
The author describes the history of information theory （before it was coined） starting with Alexander Bell， Morse， Binary / Boolean。 He explains in a very straight forward and retail manner， concepts such as ergodicity and the units of measurement known as bits and the mathematics behind calculating bits per message。 Information theory is a hammer that allows almost any discipline and topic to be a nail。 In the 50+ years since this book was written， information theory has been applied to economi The author describes the history of information theory （before it was coined） starting with Alexander Bell， Morse， Binary / Boolean。 He explains in a very straight forward and retail manner， concepts such as ergodicity and the units of measurement known as bits and the mathematics behind calculating bits per message。 Information theory is a hammer that allows almost any discipline and topic to be a nail。 In the 50+ years since this book was written， information theory has been applied to economics， evolution， medicine， and remains the backbone of computing。 Some argue it is the most robust unifying theory we know。 Great introduction for those wanting to learn about this very important concept。 。。。more
Ben Gutierrez
，
I meant to buy this years ago when I saw it on the shelf at Borders（！） and I regret not doing so。 This is another great example of technical writing that moves seamlessly between highly mathematical and conversational tones。Proofs aren't included here， opting instead to explore a breadth of applications。 After building the foundational concepts and vocabulary， Pierce leads us into relationships between information theory and physics， psychology， and even art。 I was surprised at the how well the I meant to buy this years ago when I saw it on the shelf at Borders（！） and I regret not doing so。 This is another great example of technical writing that moves seamlessly between highly mathematical and conversational tones。Proofs aren't included here， opting instead to explore a breadth of applications。 After building the foundational concepts and vocabulary， Pierce leads us into relationships between information theory and physics， psychology， and even art。 I was surprised at the how well the discussion about computers held up thirty-five years later， but then， this isn't just any pop-science writer: Pierce named the transistor and led the team that first built it。If you want a mathematical basis for understanding how communication works and the limits of what is possible， this is an excellent starting point。 。。。more
Roberto Rigolin F Lopes
，
We are in 1980， Pierce is showing off Information Theory to layperson like myself。 You may very much appreciate his introductory discourse on mathematical models and "informed ignorance" （his is humble but setting the boundaries of Shannon’s work）。 The text is warm and he dares to knit together language， cybernetics， psychology and art。 Low entropy everywhere and exciting sometimes。 We are in 1980， Pierce is showing off Information Theory to layperson like myself。 You may very much appreciate his introductory discourse on mathematical models and "informed ignorance" （his is humble but setting the boundaries of Shannon’s work）。 The text is warm and he dares to knit together language， cybernetics， psychology and art。 Low entropy everywhere and exciting sometimes。 。。。more
Sungjoo Ha
，
Beautiful explanation of various concepts related to information theory。 Instead of seeking to be technical and rigorous， the author aims to deliver a good intuition behind solid ideas。 Worth reading even though some of the discussions at the end of the book is outdated。
Muhammad al-Khwarizmi
，
An interesting， witty， mostly clear， albeit now somewhat dated look at this topic。
Lorraine Weis
，
An interesting introduction to information theory， with philosophical discussion of language and data。
Randy
，
This book is a classic。 It describes "Information Theory" as handed down by Shannon， in that respect it is above being dated since it is merely right。 If you want an understandable description of information theory with no hype （the book was originally written in the 60s）， this is a great book。 The author was responsible for the images transmitted back from Jupiter and other deep space probes， so he is both authentic in his excitement for what Shannon discovered， and for what it can do for anyon This book is a classic。 It describes "Information Theory" as handed down by Shannon， in that respect it is above being dated since it is merely right。 If you want an understandable description of information theory with no hype （the book was originally written in the 60s）， this is a great book。 The author was responsible for the images transmitted back from Jupiter and other deep space probes， so he is both authentic in his excitement for what Shannon discovered， and for what it can do for anyone working with real signals in real noise。Pierce’s musing on information theory and psychology and art are interesting， but he ends with a chapter reminding the reader of the limitations of information theory proper。 I suppose in the 1960s when the book was written， people might have sprinkled “information theory” and “cybernetics” over anything they wanted to make sound modern， much in the way artificial intelligence is used today。If you replace "cybernetics" with "artificial intelligence" or "deep learning" and transport yourself to 1980 when the Dover edition was released， （remembering that the space shuttle used IBM 360 computers with magnetic core memory）， then the author’s warnings about hype seem prescient。Pierce gives a very accessible description of Information Theory and how it is applied it to real problems in communications and signal processing。 。。。more
Aaron
，
What exactly is an introduction？ At what point do you say that a book is far too technical and the material is better fit for people with intermediate experience？ Of course， it all depends。 With that said， I think this book does still qualify as an introduction to information theory， but it really pushes the limit。 Perhaps another way to say it is that this book is better fit for students in a college course， not casual readers with a passing interest in information theory。 I definitely found it What exactly is an introduction？ At what point do you say that a book is far too technical and the material is better fit for people with intermediate experience？ Of course， it all depends。 With that said， I think this book does still qualify as an introduction to information theory， but it really pushes the limit。 Perhaps another way to say it is that this book is better fit for students in a college course， not casual readers with a passing interest in information theory。 I definitely found it interesting and I got a lot out of it， but I also consider this one of the tougher books I've read outside of my field。 In total， it took me about 18 months to read the first 180 pages （while waiting on my clothes down at the laundromat）， until I decided a couple weeks ago to just suck it up and finish it。 In the beginning of chapter 6 Pierce acknowledges the difficulty of the first chapters as necessary preparation for the rest of the book。 Those first chapters are dense， technical， and boring。 Probably too boring for the average reader。 The sheet amount of mathematical notation was just made it so much more intimidating every time I picked it up。 Eventually I decided on the strategy of ignoring whatever I didn't understand on the first pass， and that seemed to work fine。 Pierce explains the ideas thoroughly， and the equations provided just offer a useful restatement of the same ideas for the mathematically inclined。 I wish I'd known that when I started。 He even says exactly this in the Appendix （"On Mathematical Notation"）: "The reader will find a fairly liberal use of mathematical notation in this book， including a number of equations。 This many incline him to say that this book is full of mathematics。Of course it is。 Communication theory is a mathematical theory， and， as this book is an exposition of communication theory， it is bound to contain mathematics。 The reader should not， however， confuse the mathematics with the notation used。 The book contain just as much mathematics and not include one symbol or equality sign。"So there， do your best and get what you can。 The above quote also illustrates his writing style。 I wasn't sure what to make of it at first， but it grew on me。 He has a dry wit， which I enjoy but I know so many people will hate。 If you have a passing interest in communication or information theory， you might want to start off with a more accessible text。 If you have a fairly strong background in math or you're willing to make up the difference with effort， then this is probably the best introductory book on the topic。 。。。more
David
，
A gentle， but someone dated （1961） introduction to information theory。
J。
，
Maybe more of a 3。5。 A solid not-very-technical introduction， which also explores the relation between Information Theory and other disciplines。 Well-explained and easy to read。
Brian
，
Good recap from my RF engineering days where Nyquist and Shannon's work were often applied to sat-com 。。。I was hoping for a bit more on the metaphysical front but that was probably unrealistic given the chapters listed in the TOC。 Good recap from my RF engineering days where Nyquist and Shannon's work were often applied to sat-com 。。。I was hoping for a bit more on the metaphysical front but that was probably unrealistic given the chapters listed in the TOC。 。。。more
Kathleen Fredd
，
I learned a bit and enjoyed the read。 Another reviewer said it was a 'gentle thorough' introduction to the topic。 I can't speak to the thorough， not my field， but it was gentle。 Pierce has a sense of humor， and by golly， I believed that I could do the math if I took a real course in information theory。 What made it truly interesting to me was the date of revision， 1980， before the big revolution in IT。 In 1990 I still had to create a program in order to use a program。 Reading this book was a bit I learned a bit and enjoyed the read。 Another reviewer said it was a 'gentle thorough' introduction to the topic。 I can't speak to the thorough， not my field， but it was gentle。 Pierce has a sense of humor， and by golly， I believed that I could do the math if I took a real course in information theory。 What made it truly interesting to me was the date of revision， 1980， before the big revolution in IT。 In 1990 I still had to create a program in order to use a program。 Reading this book was a bit nostalgic。 。。。more
Nik
，