Reviewing The Empire of Illusions, Part II

The second illusion that Hedges takes on in The Empire of Illusions is “illusion of love.”  Put simply, it’s all about porn, or a rather puritanical rant against it laced with modern liberal sentiments.  I wonder if this is the product of the juxtaposition between Hedges the divinities student and Hedges the liberal journalist.  He repeatedly hammers home the familiar point that porn is evil because it degrades women (and other objects of pornography) to simple commodity to be consumed and not fully-fleshed out humans.  It is in this chapter where Hedges is at his most shrill, predictable, and even annoying.

Pornography and, in general, sex is an ancient trade.  That it should be a huge industry today should hardly be a surprise.  In a sense, pornography is also honest:  it sells the physical enjoyment of sex, even if in crass and unsavory forms.  It does not pretend to be anything beyond what it says on the label.  If you are looking for a meaningful relationship, you do not turn to pornography, nor does pornography sell itself as a substitute for such. What is so illusory about pornography?

A much more pertinent story about an actual illusion of love, people semi-willingly falling for them, and those who cynically exploit them for profit showed up in a  story on This American Life some time ago.  Stories of unrequited love, sustained by self-imposed illusions and consequent delusions, for its part, are also an ancient tale, the stuff of great and not so great literature since time immemorial.  Everyone in their poetry class heard about Petrarch and Beatrice, after all and how that illusion gave rise to an entire class of poetry, the sonnets.

Is Hedges just being pissy, ranting on pornography the way old feminists have?  There are signs that, up to a point, this is indeed true.  But, at the same time, he is seizing on something about how technology reshapes the data ecology (again).  Some forms of information have reproduced far more rapidly in the technologically connected age than others skewing people’s perception of the reality:  porn is definitely one of them, as is internet gambling.  Meaningful relationships are far more difficult to establish, and this difficulty, coupled with the ease of “physical” communication, has produced the situation like that found in the This American Life story.  Heck, how many stories do we encounter where people engage in make-believe pseudo-sexual contexts online, pretending to be what they are not–and in full knowledge that the other party is not who they are pretending to be?

This phoniness in the kind of human contact that we have is pervasive beyond just sexual or quasi-sexual relationships.  In 1970s, members of Congress frequently made trips to their home districts.  Many voters knew their congressmen beyond their names, party affiliations, and policy positions.  They were their neighbors and friends with whom they could chat over coffee at a local diner.  It is easy to exaggerate the extent of such personal contact between members of Congress and their voters:  we, as people, were rarely so engaged in politics that such large numbers of people would go out of way to chat with elected officeholders for coffee:  perhaps 4-5% of the voters might have semi-regularly met their congressmen, enough to know them as something more than political ciphers.  These encounters, perhaps, were not so truthful either–the appearance of humanity that the congressmen showed may well have disguised their true political agenda, for all we know.  But then, we go back to the questions of what is and is not illusion and what is “true” knowledge.

What is indisputable is that the “illusions” of the old, whether in love or politics, if that indeed was what it was, was closer and personal.  These were products of repeated interactions that required investment in time and patience.  We no longer have this.  We have plenty of information, but little in terms of the subtler insights arising from appropriate subsets them.   We do not know our neighbors, our representatives, our lovers, because we don’t have the time (and they don’t have time) to talk to us at length.  Absent the prolonged interaction, we simply pay for what we want to be delivered instantaneously in a commodified fashion, whether it is sex, or an approximation thereof, or a bill (or an approximation thereof).  Once the goods are delivered, we are unsatisfied because that is not what we really wanted and think it fake, which, in some sense, it is.  But when all interactions are reduced to commodified relationships where mere goods and services are exchanged for a payment, without a sense of trust, “intimacy,” trust, and emotional connection, what else can one expect?

This is a deeper problem than what Hedges is getting at, also of a much longer standing.  We see traces of this in literature going back centuries, especially to times of great turmoil and change:  Jane Austen, F. Scott Fitzerald, Ernest Hemingway, Tom Wolfe, just to name a few.  This was why the masquerade was the big deal in 18th century, where people could, literally, hide behind the mask to interact among themselves while abandoning the requirements of the social norms.  Material goods, prestige, or whatever else that the era seems to value becomes the target for pursuit by the characters to the exclusion of personal fulfillment who invariably wind up mired in tragedy–even if the tragedy is an indulgent (and ironic) one, like Tom and Daisy Buchanan.  Yet, we don’t have the time for personal fulfillment.  The requirements and obligations we have give us little opportunity thereof, and, even if we are fortunate to find it for ourselves, those whom we want to engage may not have the same chance.  So we settle for easy, but phony relationships:  we hang out with the people who are predictable, conditional on what we know of them, and, where applicable, pay for commodified “relationships”  that we know to be phony.

Hedges is right that, once the relationship becomes commodified, the “other,” those who offer services in return for money (whether it is sex or not) are cheapened.  They are no longer people.  They are stuff.  If we are entitled to get what we pay for, their human frailties become unacceptable.  We paid good money for it, whatever it is:  we demand our stuff, damnit!  We lose empathy and understanding in course of a transactional relationship.  Or, we become delusional in the opposite direction:  we know we are getting phony services, but since we need a sense of relationship, we insist that we are getting that connection regardless, like the men featured in the This American Life story:  yes, we know it’s phony, but it felt real to us and that’s all that counts.  But it is still merely a transactional relationship from the other side:  it is still a tragedy of delusion regardless.

The illusion of love, then, echoes the illusion of literacy.  It is not that we don’t want an illusion, for everything is, in form or other, an illusion, some make-believe that we need to invest in time to make work.  We want an illusion that we can all believe in, an illusion that lies like the truth, to all of us at the same time.  Yet, we cannot, or perhaps, do not want to invest the time for creating and maintaining that illusion.  We are simply unhappy that our money can’t buy that illusion and it is this frustration that Hedges seizes upon.


Assaulting an Empire of Illusions, by Piling up Even More Illusions–A Review, Part I.

I did not read Empire of Illusions by Chris Hedges when it first came out.  His writing strikes me as overdramatic and reactionary, the kind of screed you’d expect from a cranky old man who sees kids misbehaving on the streets.  He has a point or several worth thinking about, but his angry screeds–which is what they are–always seem misdirected and misshapen.

The Empire, as it turns out, fit my expectations exactly.  It is a misconceived screed from a bitter and angry man.  But he does have a point and the kids are indeed misbehaving badly–just not the way he thinks they are.  In so doing, he assaults this empire by building a straw alternative of another illusion, some “grand old day” when things were not so phony, which, obviously never existed.  But is today’s phoniness different from that of yesterday?  I think it is, but the difference is much more subtle than what Hedges would have you believe.

The first illusion Hedges tackles is the “illusion of literacy.” In so doing, he confounds two separate phenomena that many seem to associate with each other without justification:  Americans are not particularly well-read and they subscribe to a lot of lowbrow fakery.  As the consequence, they are unable to distinguish between “reality” and “illusion.”  He starts the book by drawing attention to a professional wrestling match, presumably the ultimate examplar of the phony that nobody calls out, followed by an uncomplimentary discourse on reality television and celebrity and phony unfulfilled promises of politicians reduced to mere slogans, interspersed by snippets of how “functionally illiterate” so many Americans (and Canadians!) are.

But is this new?  Lowbrow fakery has always been there, at the heart of popular entertainment in any era.  Professional wrestling has been around for decades, Marilyn Monroe was neither blonde nor dumb, Performers in minstrel shows were not black, among other things.  Politicians have always resorted to sloganeering and symbolism, hiding the cynical and manipulative inner  layer.  All successful politicians, whether Ronald Reagan or Franklin Roosevelt, left a long trail of disappointed and disillusioned assistants who worked closely with them and in so doing discovered the disjunction between the illusion and the reality.

Symbols emerge in public discourse because people cannot process the full breadth and depth of complex information.  As such, it does bear some linkage to the degree of literacy:  it is indeed more difficult for less literate people to process more complex information.  But it is a matter of degree, not substance.  Nobody really understands the quantum mechanical foundation of gravity in full.  Nobody understands the entire molecular basis of evolution at the cellular level.  Most popular discourse of science, or indeed, any other topic is buried in ignorance and wrongness.  In this sense, almost everyone everywhere is “functionally illiterate,” incapable of speaking intelligently about nearly any topic of sufficient complexity.  Might it be better that they could speak “intelligently” about more esoteric topics?  Perhaps, but at what cost?  Will everyone be willing or even able to spend decades learning the complex minutiae at the depths of the domains that they have no reason to wade into?  Will it be the best use of their time and productivity?  The constraints on the depth or the shallowness of understanding, then, come not only from the basic literacy but also from the time and mental resources that people have to spare.

Is reliance on symbols problematic?  All symbols are “illusory” since they leave out a great deal of information.  Symbols operate by condensing complex universe into bite-sized pieces that can be easily digested by their consumers.  There is always a “catch” to any symbol, the unspoken, unobserved underside that elude their audience.  It is, at its core, a statistical problem, of crafting a model to make sense of a given set of data.  Within a particular set of data, you trade off between bias and variance in constructing an estimator:  you can have a complex estimator that minimizes bias at the cost of a higher variance, or you can accept a greater bias while lowering the variance.  It is noteworthy that statisticians (and data miners) are increasingly willing to accept a biased estimator in favor of smaller variance, in order to gain greater, more subtle insights from smaller samples.  Even more ironically, this is taking place as the amount of data is growing larger because of the data ecology problems:  the less informative data reproduces far faster than the informative and the demand for information expands at a comparable rate as the whole sample, rather than the subset of the useful data, it seems.  A useful illustration can be drawn from searching for a given piece of news about almost anything these days:  googling for the news item yields hits that mostly repeat the same story.  Google’s search algorithm itself conspires to produce this, because it will systematically ferret the information that “people want to see.”  If you are looking for a different piece of information, from an unconventional or a contrarian perspective, it will be buried under a mountain of conventional wisdom that will take hours to dig out if at all possible.

What is the information environment that people face today like?  On the whole, information abounds:  there is information aplenty about everything and anything.  But the subset of “useful” information on any topic is rare and difficult to come by.  People buy into the “illusions” more than they did for much the same reason data miners delve into low variance-high bias estimators:  in order that they can extract more from the little data, giving rise to the paradox that, in the midst of informational plenty, we resort to the tools of the informationally starved.

Inference from a small sample size is fraught with another problem–if one could characterize it as a problem.  The rate at which a learner “learns” from a set of observations is the function of his or her prior distribution–that is, what the learner believed the data distribution in the universe looked like even before seeing the first datum.  I noted in an earlier post how experts process information different from the laypeople:  the experts already “know” what the universe looks like so that they place the new information (or, “update their prior” in the statistical lingo) differently from the laypeople.  But, in the realm of statistics, the “knowledge” of the experts is not privileged.  It is simply another “prior” that the learner brings to the table that provides context for the new information.  Not all laypeople will bring the same priors.  Different priors will react differently even to the same information.  With a sufficiently prolonged exposure to the same stream of information, all posterior distributions converge, regardless of the prior.  Not so when the new data are scarce:  priors dominate.

This brings us back to the problem Hedges noted as the “illusion of literacy,”  that people today think they know far more from very limited information, that they are filling in what they don’t know with their own existing beliefs, predilections, and prejudices much more than they did in the past and mistake that as the reality itself.  Put differently, symbols count for far more than the new information about the reality.  This applies to both fans and haters of the symbols.  Is Barack Obama any less phony than Franklin Roosevelt, or George W. Bush more of an illusion than Ronald Reagan?  I do not believe so.  The kind of far reaching conclusions that so many seem to jump to about both of these politicians–or, indeed, any other celebrity–based on trivial amount of information is astonishing.  Yet, with so much information about all manner of things to wade through in today’s data ecology, acquiring and processing anything deeper is difficult.  We buy more into illusions because finding alternatives–and the time to make sense of them–is more difficult than ever.  Because different illusions, coupled with shallow information streams, are processed differently, every group of learners reach different conclusions.  Many can see the others subscribing to idols, because they lack the priors with which to jump to the like conclusions quickly–except for the symbols they themselves prefer.  Whereas all but the deepest insiders bought into the illusion of FDR or a Marilyn Monroe, only a relative few buy into the illusion of Barack Obama or Angelina Jolie.  The others see them as illusions that they are–for these are not their gods.  The awareness of the phoniness of the world abounds, but without a consensus as to what the “truth” is.  Such consensus, of course, cannot be built around the real “truth.”  We don’t know what it is.  If we did, we wouldn’t need science.  People need “truth” as an anchor, something that they can believe in and trust.  The real truth fails at it miserably.

Again, not a brand new phenomenon.  In the 4th century CE, Emperor Theodosius had a solution to this problem:  the official illusion accepted and enforced as the universal truth by the sword of the state.  Like it or not, it worked:  it created the Western Civilization as we understand it.  “Politically correct” orthodoxies from all sides, whether they be the multiculturalist-cosmopolitan vision of the so-called liberals, unregulated untaxed libertarian universe of the so-called conservatives, or the good old days of the populists, are all vying to be the next Theodosian truth.  As it must be “universal,”it cannot coexist with the alternatives.  All are equally illusory, and the skeptics–whether they are as jaded as Chris Hedges or myself or simply devotees of the other illusions–can see it and they outnumber the believers.  The sword of the state, with which to enforce the universal truth over the unbelievers, is equally lacking.   Emperor Theodosius, or his modern equivalents, however, might say that the truthiness of the illusion is not the important part–but that the illusion exists universally and that most people buy into it enough to form their behavior around it.  As Fred Brooks observed in The Mythical Man-Month, the mark of successful leadership is not that it makes the “right decision,” for what the right decision looks like cannot be known a priori, but that it makes decisions that the whole organization can work around, to act in a coordinating capacity rather than a “factual” capacity.

This is indeed a troubling realization, if the goal is to seek out the “truth” at all costs.  Dostoevsky’s Grand Inquisitor would approve.

Reviews on the other five illusions to follow in coming days…

Data Ecology, The Haymarket Square Controversy, and Learning from Data

Some years ago, there was a big controversy on wikipedia over the Haymarket Square bombing.  Basically, some guy was trying to edit the page to introduce notions different from the conventional wisdom, contrary to the rules of Wikipedia, and the guy was repeatedly slapped down.  The story received much attention at the time (see for example, this article on The Chronicle of Higher Education.)

If you cheated and looked at the article, of course, you’d have noticed by now that the one troublemaker was in fact a professional historian and one of foremost experts on the the topic and his research showed that a lot of conventional wisdom widely believed among the public is wrong.  The problem is that this additional piece of information is not known in the anonymous internet world:  everything is datum of equal worth, without knowing what lies behind the information.  The commonly available information and “the truth” are the same, then, without the means to discern the truthfulness of the information.  Now, here is where something gets tricky:  does not Wikipedia have rules on “verifiability”?  Yes, but only based on secondary sources:  primary sources are not allowed as source of information.  In a way, there is a good rationale behind it:  primary sources are invariably of dubious credibility and evaluating the information they convey is not easy for the lay audience.  That is where outside expertise and “secondary sources” come in:  the information is valuable because such and such “expert” said so.

But how do you evaluate the worth of the secondary sources?  In many instances, they themselves perpetuate myths and wrongheaded conventional wisdom.  Evaluating how “accurate” and “wrongheaded” they are itself requires nuanced expertise.

The original Encyclopedie edited by Diderot (and other early attempts at creating encyclopedias) understood the value of expertise that goes beyond quantification.  They sought to bring in input from the foremost experts of the time.  They did not rely on the presumed wisdom of the big anonymous crowd but on the weight of academic reputation of the few highly renowed experts.  Champions of wikipedia and such might claim that they might have done otherwise had the technology been available, but it is highly doubtful:  the intellectuals of the Enlightenment were, quite frankly, snobs.  They did not believe that the crowd had much wisdom to offer.  Appeal to expertise was a very deliberate choice unconstrained by technology.

There is something in this story finds an echo in the mindset of “data mining,” at least the naive approach thereof.  The foremost concern of data miners is to identify patterns in the data, not necessarily understanding the data itself, theorizing about its distributions and properties, and spending much time concerned with how the data was generated in the first place.  (Although many statisticians also had similar weaknesses in the past, one might add–and may still do.  They sought to justify their beliefs, say, about human intelligence, using data and statistical techniques, rather than trying to understand the nature of human intelligence itself, where their data was emanating from.)  The problem is that data that is available to be analyzed exists in the quantity that they exists in for a reason.  This is, in part, an extension of the simple selection bias problem, but exacerbated by the nature of the data ecology today.  Data is not simply plucked out of the “reality.”  It is plucked from both the reality and its many echoes.  The “bad” data that fails to yield useful insights are not only abundant, but they generate far more echoes than the good, useful ones.  In other words, “bad” data outreproduces the good ones exponentially, if one were to conceptualize data availability in ecological terms.  The insights that can be garnered from the “bigger data” are potentially misleading because they are drawn far more from the uninformative but more abundant data.

The challenge here is the same as the problem that wikipedia has not yet addressed–as far as I know–about such things as the Haymarket Square bombings.  In absence of the subject knowledge (or domain expertise, in data science lingo) that permits appropriate weighing of different sets of data, properly evaluating the data is difficult.  But what makes wikipedia and naive machine learning valuable is precisely because they lack the “domain knowledge” to begin with, for along with knowledge comes prejudice.  People “know” what conclusions to draw without insufficient evidence either because they “know” or “think they know–but really don’t,” like Karl Pearson about human intelligence.  In an ideal world, the subjects experts should be skeptically respectful of the prospects offered by naive pattern recognition, or the data miners should be equally skeptically respectful of the subject expertise.  But being “skeptically respectful” may well be the most difficult state of attitude towards information to attain.  The data miners, for example, have tons of data that say one thing.  They need to know, in terms that they can understand, why the patterns they are seeing are misleading.  Simply shoving the subject credentials does not–and should not, for the sake of advancing knowledge–impress them.  But the data analysts need to be cognizant of the limits in the data, the data itself as well as its statistical properties (there will be another post in near future about why taking variances seriously might be a good idea–perhaps even more so than the point estimates.).


I just came across this blog post.  In some sense, this reflects what I had just noted, but draws the opposite conclusion.  The consensus of “knowledge” is inherently conservative and new ideas, even if they are right, always must fight for its place by displacing the existing champion.  Many historical accounts of the Galileo affair noted this:  while Galileo was ultimately vindicated to be closer to the truth than his opponents, he did not have either convincing evidence or theoretical explanation that could overwhelm the consensus among his scientific peers.  He was put on trial (and given very light punishment) mostly because he was a foul-tempered crank who slandered the important figures of the day, not because he was challenging some immutable (anti-)scientific orthodoxy. In some sense, even if the scientific consensus was wrong, the inertia of incredulity Galileo encountered was not atypical of radically new scientific insights.

The problem is that Galileo, and for that matter, experts in general, deal with other experts who are already in possession of a great deal of expertise.  They do not need to be told of the first principles and how they lead up to the conclusions.  Galileo’s contemporaries were familiar with the Copernican theory, for example, and were deeply appreciative of it. They were not wrong thinking that the weight of evidence at the time was limited, however.  The necessary information needed to tip that balance was small.  This is what makes the “wikipedia vs. the truth” problem more troubling to me:  it punishes expertise of the few and perpetuates the ignorance of the masses, in the name of “democracy.”  Tipping the balance of the non-expert beliefs will take much more.  Tipping the balance of non-expert beliefs seemingly backed up by copious data will be even harder, unless Landon loses in a landslide to FDR. (NB:  The “Landon Landslide” story is far more pertinent than one might expect:  The Literary Digest drew its conclusion from exhaustive analysis of very large data, consisting of several million respondents.  Its polling prowess was supposedly validated by successfully forecasting several previous elections.  The sampling problems that bedeviled the Literary Digest polls produced publications in public opinion research decades after the fiasco.  In the same election, George Gallup proved his mettle as a pollster not by his data analytical prowess but by his awareness, drawn from his political savvy, that the data ecology in 1936 was dramatically different from the previous elections.  To the degree that one wishes to make accurate predictions in the sample that you do not have yet–i.e. NOT your “testing” data–it may be worthwhile to focus on how to forecast changes in the data ecology, not ingenious ways to cross validating by naively cutting up the data that you have…although cutting up the data on hand may actually prove to be the best approach to evaluate how the model holds up vis-a-vis potential evolutions of the data ecology.)

Conditional Variance?

Mark Twain supposedly said “It ain’t what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so.”  Old Sam Clemens, it seems, had much keener insights into statistics and its abuse than he gets credit for.  (along with his other quote, “lies, damned lies, and statistics.”)



Let us think about that…while I go about organizing my thoughts….