State of the Economy and Trump

A friend of mine wanted to know the correlation between Trump’s primary performance and the state of the economy, in light of the recent report from the Census Bureau that the median income in country has gone up sharply between 2014 and 2015.  The report itself notes that the growth was uneven, though, with rural areas being left out of the growth spurt, as per Nate Cohn’s tweet.  If so, could those left out of the economic growth be voting for Trump?

Some problems arose:  while the report has been released, the official data has not been. But, even as the magnitudes might differ, the patterns of who is gaining and who is losing would not have been too different from 2013 and 2014, for which county by county data was available.  Secondarily, however, Trump was winning everywhere after he became the obvious winner, making the results effectively useless after some time.  So we needed to impose some semi-arbitrary cutpoint based on the primary calendar, for which we settled on two–keeping only the Super Tuesday numbers and keeping only the data through Ohio primaries.

The first, the Super Tuesday primaries, yields the following plot:


The unit of analysis is county.  The X-axis is the change in median household income between 2013 and 2014.  The Y-axis is the percentage of votes won by a candidate on Super Tuesday, among the Republicans.  The dark blue line is Trump:  he is the only candidate who did worse in the counties where the median income increased.  Cruz, indicated by the green line, did much better in the counties where median income increased.  Rubio, indicated by the light blue line, just didn’t do well anywhere.

Expanding the sample to all primaries through Ohio, but keeping the same candidates, changes the pattern slightly, but only slightly:


Trump is still doing better in counties had worsened between 2013 and 2014, as measured by median household income.  There is no discernible pattern for either Cruz or Rubio, however.

When all the primaries are included, no discernible pattern remains (Rubio is dropped from this graph).  trump-vs-gopers-all

Trump could be doing better in counties where income dipped, or not–with confidence intervals like these, who can tell?  The story conveyed by these graphs, then, is that Trump’s early supporters indeed did come disproportionately from the counties that suffered economic downturn recently (whether these voters individually did suffer an economic downturn), but were increasingly joined by more economically fortunate Republicans as he became increasingly likely Republican nominee.

Certainly, these are based on limited data and constitute only highly tentative suggestions.  But it is worth noting that, even in times of prosperity, there are those who lose out economically, while even in times of economic privations, some people do make out like bandits–the whole point of focusing on the variances, rather than means, indeed!  In an era of economic polarization, a positive change in the mean (or even the median), even a large positive change, does not necessarily mean that all boats are rising equally. Many may be sinking and they are discontented enough to seek and demand redress, and this may well be a significant chunk of the force behind the Trump phenomenon (Interestingly, this pattern is not replicated among the Democrats, where the pattern is messier.  It does seem that Sanders generally drew better among the higher income voters, or at least in the counties where they reside, ironically.)

The evolving pattern of the Trump coalition over the primary season should worry the Clinton camp:  even as the Trump coalition was built on the economically precarious from the beginning, he did draw in a lot of affluent Republicans eventually.  The MWV’s, the economically precarious lot among the electorate who are not minorities plus the affluent, more customary Republican voters do equal a majority, albeit a small one, in both electoral and actual sense.  As I keep noting, Trump seems weak because he has trouble with the Republican voters, especially women.  If, as I suspect, that Clinton will not be able to draw many of them to actually voting for her, or even sitting this election out, and cannot inspire the turnout rates among the minorities as Barack Obama did in 2012, it is not too improbable that she’d lose this election.

But this is more than just about this election.  An inequitable distribution of economic fruits begets social and political instability, if only in form of voter revolts rather than peasant or worker revolts.  As Bismarck would have said, he had been given to more humor (and writing in English!), what use are money and power if you are constantly in danger of losing it to the chaos you are creating by grabbing more of either?  In politics (and political economy, as all matters of economy must be eventually) stability comes first.  Disruptions may make you money for now…but will take everything away soon enough if you are too reckless.



Polling Trends and Electoral College

Two interesting tweets that need expansion on:

First, from Nate Silver:

As others have mentioned, there are echoes of Brexit here. Clinton’s narrowing lead over Trump a lot like Remain’s over Leave at this point.

This is rather meaningless, in terms of data, but an informed hunch.  Still, “informed” hunches are often more informative than data, in the right hands.  We run into situations that never happened before every now and then, and we know that the situation tomorrow will not be the same as that today–even if we only have data from today.

We can, however, reason beyond hunches:  Britain is far more homogeneous a country, in both ethnic and “cultural” terms.  There is far greater consensus about what Britain means to the English, at least (the Scots are a different matter), who, in turn, make up a far larger proportion of the population.  In the United States, this sort of consensus is not nearly so strong.

Then there is this from Steve Saideman:

@NateSilver538 how did the electoral college work out in the UK?

I think this is actually where Trump could gain an advantage, even if the numbers may not be in his favor.  Most minority voters are concentrated in not exactly competitive states–CA, TX, NY, etc.  The competitive states, mostly in the Midwest, are whiter than deep Blue or Red states.  Things could be interesting, with regards minorities, if NJ or TX were to become competitive–which may happen, more likely this year than others, but probably won’t.  Trump’s support among the working class whites pads his advantage in the states that matter more because of the electoral college, negating some of the disadvantage he incurs because of the differences in UK-US demographics.

Poststratification, or Why Look at Crosstabs.

Andy Gelman has an excellent post that almost constitutes a direct salvo in defense of the USC polls being conducted for LA Times.  (NB:  I think USC polls are still skewed somehow, but there are also important bits of information being lost in the usual polls that it provides).

The truth about elections and parties is simple:  party id’s don’t change too much and people tend to vote party.  If a poll shows a big swing in one direction or another, accompanied by a big swing in the partisan composition of the respondents, then the poll is probably capturing a big swing in survey response bias that correlates with partisanship.  We want to know what the choice will be, conditional on all sorts of factors known to be correlated with  the vote choice–and partisanship is indeed highly correlated with the vote choice.

In context of 2016, this is especially relevant since Trump’s unpopularity is especially acute among the Republicans–at least, those voters who are reliably Republican in most elections:  wealthier white suburbanites.  He is barely tied with Clinton among this subsample, according to various polls, and trailing significantly behind among the women among them.  These are, in a sense, far more significant predictors of Trump’s defeat than how unpopular he is with the minorities, who tend to be overwhelmingly Democrats anyways.

Whither Parties?

I think Corey Robin is drinking too much of his own Kool-Aid.

The problem with the Republican Party is a fundamental problem affecting the root of party politics, especially the way it has evolved in the past few decades in United States, not anything related to “conservatism” or any other ideology.  It is rooted in institutions and the same problem, albeit of less extreme variety for now, is bedeviling the Democratic Party.

The so-called Tea Party in the Republican Party actually consists of two distinct movements that have gotten conflated, both by their own politics and by the outside observers.  They are, in context of 2016 elections, the Ted Cruz Tea Party and the Donald Trump Tea Party.  They share the same trait:  they are outsiders to the traditional Republican machinery.  The crucial distinction is that the Ted Cruz Tea Party has a fairly clearly defined political agenda and is interested in capturing the machinery to use it (and abuse it) to advance their own agenda, regardless of the medium to long term effect on their party or the rest of the political landscape, while the Trump Tea Party is a heterogeneous lot without a clear political aim but is characterized by deep distrust of the existing political institutions.  On the Democratic side, the history of this conflict is rather different and the conflict has been taking place much longer, but is of similar nature.

In the abstract, the power of party institutions comes in two flavors.  First, the formal powers of the institutions allow those who wield it to block adoption of the alternatives they disapprove of while accelerating and otherwise favoring the adoption of those that they do–negative and positive agenda powers, as the political science lingo labels them.  However, this power is fragile:  it can be deprived if an opposing coalition emerges with both sufficient numbers and sufficiently clearly defined aims to the contrary.  No democratic institution (and the same logic applies even to most non-democratic institutions) can force through outcomes that are actively opposed by a very large number of people, especially those with whom they share the institution–such as other members of the same party, other members of the legislature, and so forth.  The more important power of the institutions, then, is the ability of those who control it to define the conventional wisdom, or the narrative, that can serve as the focal point, to convince other participants in the political process who do not have clearly defined goals, preferences, and beliefs that they should want X, even without clear knowledge thereof, because X is the preferred alternative of “the party” or whoever.  In the much ballyhooed and now increasingly discredited (undeservingly so, in both cases) book, The Party Decides, both forces are present:  the party’s choice serves as the focal point for the many, many voters who don’t know and don’t care much, while the formal powers can be used to slap down the handfuls of troublemakers with actual dissenting agendas.

The problem for the party is that while the former, the more formal set of powers, has grown, the latter, the informal foundations on which they are built upon, has been badly degraded.  Mayhew, in 1974, already foresaw the impending crisis for the party:   rival power centers to Congress, like the president, can pursue their agenda without respect for the kind of consensus sought by the Congressional party leaders; ideological and other factions with their own agendas can publicize their aims and mobilize support for them through extralegislative/extrapolitical means; and the changes in technology was making it easier for these rivals to the traditional party politics to intrude on the leaders.  (The entire second half of the classic 1974 Mayhew book is on this topic–but no one seems to remember any of these!)

The considerable formal power of the party leadership is a draw for the factions that are not so much interested in maintaining institutions and the associated powers stable, but in using them to actively pursue their agenda.  The first serious civil war over this in the Republican Party took place in 1990s already, that pitted Newt Gingrich, who, despite the reputation he acquired as the Speaker, was actually interested in building a long term power base for the Republican Party on broad consensus (every one of the Contract with America items drew support from a majority, or at least, a very large minority, among the Democratic members of Congress, after much wheeling and dealing behind the scenes) against Tom DeLay, whose attitude towards the power might be summed by paraphrasing the quote attributed to Madeline Albright, “what’s the use of all the power of the party leaders if we don’t use it to aggressively advance our ideological agenda?”  Needless to say that DeLay won and this set the pattern for the rest of the GOP:  the ideological faction should actively seek to capture power, use it to advance their agenda aggressively as well as to beat down their intraparty rivals who get in their way.  The Ted Cruz Tea Party is the natural progression of this attitude:  the Republican Party is useful for them only so far as it can be used as a tool to implement their ideological view as policy.  Among the Democrats, the same civil war took place much farther into the past, in the guise of “House reforms,” where the liberal wing took power and purged the Old Guard who were interested more in maintaining internal balance within the Democratic Party.   The conventional wisdom holds that this “strengthened” the Democratic Party.  This would only be true if parties were to be viewed solely as the vehicle for making policy, in much the same manner as DeLay and, later, the Ted Cruz Tea Party conceive parties to be, with the most minimum of winning coalitions.  If the parties are to be viewed as a vehicle for maintaining balance and stability, this was a crippling blow that contributed to the poor electoral fortunes of the Democrats later.

The other Tea Party, the Donald Trump Tea Party, exists on a completely different plane.  It is not made up of ideologues who are particularly interested in implementing a particular program, assuming that they are at all interested in the program.  It consists of those who are justifiably suspicious of those who control the machinery of power, who feel that their interests, even if they cannot articulate them clearly, are not being taken into account by the powerful who are too busy with their own agendas and shutting out all their rivals by using and abusing the formal powers that they control.  Ultimately, it is a matter of trust–we don’t know what exactly we want, we don’t know what exactly they should be doing that they are not, but we know that these guys are not our friends and are looking to cheat us at every opportunity.  And they are right, for the ideologues have no interest in wasting time on those who cannot help them achieve their policy goals.  To the degree that the Ted Cruz Tea Party was mainly organized to topple the power of the incumbent leadership of the Republican Party, both “Tea Parties” made for natural allies–they shared a common enemy.  Once the Ted Cruz Tea Party, or at least its fellow travelers became powerful in Washington, they became the enemy of the Trump Tea Party, as much as the older Republican leaders.  In a sense, the increasingly narrow policy pursuits by the Ted Cruz Tea Party, made it even more blind to the discontent of the Trump Tea Party and may well have earned enmity faster.  The defeat of Eric Cantor, if not an actual member of the Ted Cruz Tea Party then certainly a close ally, by a “Tea Party-aligned” insurgent movement in the Republican primaries in 2014–supposedly a good Republican year–should have drawn everyone’s attention to the peculiar divisions within the Republican Party.   As an analogue, imagine what might happen if Anheuser Busch decided to get rid of all cheap beer in favor of expensive beer that “you want.”  Some people might pay extra for Bud Light because they like its taste, but vast majority of Bud Light drinkers who do so because it is cheap or for any number of reasons other than taste will be outraged and may never buy another A-B product again.

In a sense, Democrats already had their own version of Eric Cantor, already, in the person of Bill Clinton and the DLC.  However, unlike Cantor, the first Clinton moved the Democrats in a different direction.  By 1980s, the Democrats already had a leadership that was interested in using the Democratic Party as the means to advance a liberal agenda, and that was losing them elections–Mike Downey, a congressman from 1980s, supposedly said, “If we wanted to pass a bill that suited the tastes of the average person, we have to pass the Republican bill.”  Bill Clinton and DLC did not argue that the party should deemphasize the policy orientation in favor of maintaining stability, but that it should use its powers to pursue a different set of policy, those that, by the standards of 1980s and early 1990s, may not be so polarizing.  Or, in other words, the Republican bill that Downey was complaining about.  And, as the president, Clinton did exactly that.  For all the apparent animosity between Gingrich and the Clintons, they actually made an excellent team:  Gingrich, in his desire to build a governing party that was stable and enjoyed a broad base of support, was willing to pass legislation that suited many Democrats.  Clinton, of course, wanted to pass bills that met the taste of the average man, which Gingrich supplied.

Notwithstanding the difference in the direction of the policy, the Democratic Party today is, no less than the Republicans, a tool subservient to the pursuit of policy:  for many, the Democratic Party and the direction of the policy that it pursues are indistinguishable.  The internal struggle within the party, then, is not over whether it should be policy oriented or stability-oriented, but simply over what direction the party should pursue in terms of policy.  The folly of DLC and its legacy, for many, is that it focused on the policy that suited the taste of the “average man,” which, in 1990s, was in accord with Gingrich, not that the Democratic Party was reduced to a policymaking tool, in any direction.  The answer by the liberal critics of the current leadership is that it should pursue more liberal policy–not so much that it should stop focusing on particulars of policy questions and start listening and rethinking about how to address the unmet needs.

The catch, then, is that the direction of the policy does not matter much.  The more narrowly a party might be focused on the policy pursuits, the more likely it is to leave many of its supporters behind.  The leftward orientation of the Democratic Party in 1970s and 1980s led to the abandonment of the white working class who backed Reagan, not necessarily because these voters were “conservative” but because they could no longer trust the Democrats to be concerned over their interests.  (It is noteworthy that, vindicating Mayhew, the presidential tide turned far earlier and more decisively than the Congressional.)   The rightward turn by Bill Clinton and continued by Hillary Clinton does not change the fundamental dynamic–they simply alienate a different group of voters.  If the Democrats turn left again, the same scenario would repeat itself.  The trouble with the Democrats, then,  is exactly the same problem as that divides the Cruz Tea Party and Trump Tea Party, at least in institutional terms.  It is a divide between policy-seekers and insurance-seekers, those who want to do things and change the world in their image vs. those who need protection from the changes, including those that the former want to bring upon the world.

The Hillary Clinton wing of the Democratic Party is primarily interested in the party machinery and its associated powers as tools for making policy.  They want to know what policy they should pursue because, other than making policy and the details thereof, they have no sense of what a party is supposed to do.  Their opponents–the rank and file Democratic supporters who were unsatisfied with the party leaders, much more than the liberal critics thereof–who found their voice in the person of Bernie Sanders in 2016, do not have a clear idea of what specific policy they want to see pursued–at least going beyond some popular ideas that do not collectively make up a coherent”ideology.”  However, like the Trump Tea Party, they also know that the single-minded pursuit of policy by the party insiders is drawing them away from paying attention to their needs and interests, even if they cannot precisely spell them out in terms that can be translated to bills.  Once again, the problem is ultimately that of trust–which has been showing up in the polls repeatedly.

However institutionally analogous they might be, the discontented Democrats do not overlap much with the Trump Tea Party.  This needs to be made clear, as it is critical in shaping the electoral landscape today.  The former are, after all, Democrats, or at least, Democratic sympathizers, while the latter are Republicans or Republican sympathizers.  The relatively few real independents in the electorate might swing between the two camps, but, again, they are relatively few.  The average Sanders voter, for all the disappointment, is a Democratic-sympathizer and he will not turn.  Who might, had things been progressing differently, is a sizable minority composed of the true independents, but the prospects that Trump might lure away many of the independents who supported Sanders seem to be dimming daily, due to his own wackiness.  If the choice is ultimately that of “trust,” Trump has not exactly shown himself to be a trustworthy person for many beyond his relatively narrow band of fans.

The likely failure of Trump, however, does not obviate the inherent problems facing the parties, both of them.  The second face of power, the trust and the associated ability to act as the focal point for the uncertain partisans, is ultimately what sustains the first power.  The overreliance on the first face of power has effectively broken the latter.  When the narrative breaks, it is not easy to put it back together, without some great big myth and a larger than life founding father–an FDR, a Reagan, or a Lincoln.  All the institutional rigging to shore up the first face will not be enough–indeed, it may even exacerbate the erosion of trust and subvert the second face of power even more.  This is the real danger that faces the American party system today that goes far beyond the problems of “ideology.”

PS.  I think a simple way of describing the problem (which, incidentally describes the variance-centric thinking vs. the means-based thinking) is that the dissenting voices in both parties want someone who listens, who recognize that the answers are still problematic, not someone who has answers, even better answers.  Answers, like the means, may be right on average, but often wrong–perhaps, even wrong for everyone (e.g. the mean prediction for the number of heads of a fair coin will ALWAYS be wrong for any single coin toss.)  The important thing, rather than getting the mean right, might be to recognize the variances exist–ie. how wrong the answers are for different peoples.  This is all the more important because, when “the answers” become a narrative, like the standards of “cuteness,” the variance is significantly underestimated.  The correct answer may be .5, rather than .6, but it will still be wrong on the next coin toss.  Blaming the coin for not producing just half a head, rather than one full head or zero head will not resolve the problem.

PPS.  The great insight that Bill Clinton and DLC had was to ask, as the Democratic Party was committed to becoming a vehicle for policy, whether pursuing the policy that was against the wishes of the average man was a good idea.  Now, three decades later, his wife faces an altogether different challenge, where the average man does not know what exactly he should want any more, but does not trust the people who are running Washington to be interested in him.  The attitude taken by both parties was to exploit the average man’s uncertainty and ignorance to their advantage, further subverting his trust.  So, now what?

Variance Centric Thinking and Making Sense of Trumpism

Carl Beijer’s argument about sources of support for Trump is an excellent example of how variance-based thinking beats means-centric thinking.

For a means-based thinker, Trump’s message is about trade, racism, and other ideas that don’t make a great deal of sense.  However, since his message can be placed on a map, so to speak, it follows this line of thinking that his followers are motivated by the mean of his message that can be calculated.  Since the mean of his message lies in the realm of insanity, it follows that those who follow his message must be as crazy as well.

The variance-centric thinking does not presume to measure the mean of Trump’s message:  it is all over the place so that calculating the mean does not make much sense.  (If the distribution is very fat, you can be very far away from the mean and still catch a meaty part of the distribution.)  They key is simply that Trump is different, drawing an usual, but very identifiable following who may or may not take his proposals seriously.  The important thing is to identify those who are–and why they don’t fit the existing paradigm, rather than why they fit Trump’s paradigm.  To paraphrase Tolstoy, if all unhappy families are different, asking why they are unhappy will yield too many answers none of which will be universally true.  It makes more sense to ask why aren’t they happy–that is, what their deviations are from the formula of the happy families and start the analysis from there.  (In the like vein, the taste preference of Bud Light drinkers cannot be cleanly identified–they are a heterogeneous group who choose Bud Light either because it is cheap, they like the ads, they have no taste, they had odd taste that is not being met by anyone else, or because they genuinely like Bud Light.  If you think you can profit by identifying the “Bud Light taste” and profit by jacking up the price, you will be in for a huge disappointment.)

So the Trump voters are…different and heterogeneous, but they are similar in that they are unhappy with the status quo and are drawn to Trump for many different but understandable reasons–most of which are directly or indirectly tied to the economic and social changes.  Trying to put them all in the box, by identifying their mean and using it as a stand in for them all is bound to disappoint because their variance is so huge.

PS.  In a sense, it is really a matter of knowing the limits of your data:  can you profit by knowing the true mean of Trumpists’ (or Bud Light drinkers’) preferences?  What is the variance on their tastes, preferences, or motivations?  If they come with sufficiently small variance, then yes, you do have a lot to gain by knowing their mean preference with precision.  If they don’t, then all the extra data will only give you an erroneous belief that you know what the truth is, not the truth itself, which is inherently too variable to be pinned down.

Variance vs. Means Centric Statistical Thinking: An Illustration.

I’ve written a lot about means-centric vs. variance-centric statistical thinking.  So what do I mean by it?  In the former, we focus on the ability to “predict” something, at its mean or whatever.  In this sense, variability in the data is a nuisance, even an enemy, something to be minimized.  In the latter, however, we want to know what variables cause the biggest variability in the data.  The variance is not only an essential component, it is the main subject of our thinking.  If we can predictably forecast outcomes, it is not only boring, it is also something that can be manipulated and “pumped,” until it is broken (as per my discussion of how signals in a correlated equilibrium can be abused until it breaks, or indeed, as is the logic behind Stiglitz-Grossman Paradox, which is really just a variation on the same argument).

In the end, really, math that accompanies both approaches turn out to be the same:  in the means centric approach, you identify the predictor variables that help minimize the “errors”; in the latter, you identify the variables that happen to be correlated with the variances–which turn out to be the ones that minimize the “errors.”  This convergent evolution, unfortunately, obscures the fundamental philosophical difference between the two approaches.

An illustration might be found using some data from the 2016 primaries.  Consider the following plot.

trump and dem2012 in white and black

The graph illustrates the support for Trump in primaries as a function of the Democratic voteshare from the 2012 election, with two different types of counties:  whether the county population is above or below 75% white, which is roughly the national average–the red dots indicate the counties with below 75% white population.  The biggest variability in the Trump support can be found in the areas where Romney did well in 2012 (i.e. small Democratic voteshares):  the Republican primary voters in Republican dominated areas with large minority populations did not like Trump, while those from the counties with largely white populations did not have much problem with him.  Yes, it is true that Trump did well in many counties with both large Republican majorities and significant minority populations, but the counties where he performed poorly conditional on large Republican majorities are mostly characterized by large minority populations.  As a predictor, this is terrible:  because the conjunction of large minority population and large Republican majority from 2012 does NOT predict weak support for Trump necessarily–there are too many exceptions for that.  But, the reality is that conjunction of all these variables moving in the same direction does not happen–to pretend that they do so feeds into the conjunction paradox identified by Tversky and Kahneman, in which people think the conjunction of characteristics believed to be correlated with each other, rightly or wrongly, is also the most likely–e.g. “Linda is a bank teller and is active in the feminist movement” rather than “Linda is a bank teller.”  People already prone to believe conjunctions happen with too great a frequency already (which partly accounts for the beauty contest game–how people trying to follow a “narrative” systematically downplay the variance)!

From the variance-centric perspective, the large gap that opens up for Trump’s support in Republican friendly areas with large minority populations is not merely interesting–it IS the point.   It is the variability that we are interested in.  Incidentally, this is why Trump’s support numbers are jumping wildly–his support in many red states (i.e. the South–where Republican electoral dominance and large minority populations coincide) is highly uncertain, leading to what Nate Cohn calls “Trump’s Red State problem,” which, to be fair, should already have been apparent from the primary data already–and the polls that showed Trump’s serious national level unpopularity consistently indicated that he is characterized by particularly low popularity among the Republicans.

The key reason that this cannot be readily translated into a prediction is that we know more than the data itself, or rather, we have a broader context that includes data from elsewhere, in which to place the present data.  As Gelman et al observe, that respondents say that they voted for a particular party in the last election (in a poll) is a significant piece of information known to be highly correlated with their present latent choice, even if we may not entirely trust their response to be accurate or honest.  To insist that this be ignored is foolish–even if it cannot be taken at its face value, especially if it is correlated with a particular variability seen in the data.  To the degree that the reality is inherently complex and uncertain, coming up with a fully accounted for prediction model that can predict everything is, quite literally, in the realm of impossibility.  Much better to adopt a two step approach to learning:  identify the sources of variability, then investigate for the correlates of the variability, with the awareness that variability itself is a random variable–i.e. the variance itself may be correlated with particular variables themselves.  (NB:  homogeneity is an absurd assumption and not really a necessary one, except to make OLS BLUE, sine variance is always variable…)


On Polling Outliers

This article on NYT Upshot used to be titled “A favorable poll for Donald Trump has a problem,” I think.  Whatever the case is, it is now retitled “may have a problem,” which I think is wise.

This addresses something that’s been bugging me, albeit more generally than the USC polls: what do you do with polling outliers?

To be entirely honest, I think USC polls are doing something innovative and insightful:  the point raised by Gelman et al in this article is that accounting for party affiliation of the voters, which pollsters seem very averse to doing, is in fact a quite sound thing to do, given the stability of the party affiliation and the vote choice in today’s electoral environment.  Indeed, the way USC folk have constructed their panel essentially achieves through the research design what Gelman et al did in their study by post stratification–creating a sample that incorporates a reasonable mix of Republican partisans and Democratic partisans.

I am not so sure if the problem where people overreport in favor of the winner is as big a problem as the NYT folk make it out to be:  in the end, the past vote choice is itself a proxy, for party affiliation of the voter.  As it were, past vote choice ought to have been closely correlated with the party affiliation today, with the Republicans recalling (probably correctly) voting for Romney and Democrats recalling voting for Obama.  Unless an unexpected number of Republicans remembering voting for Obama were found in course of sampling–which would be a dead giveaway sign of the kind of overreporting in favor of the winner that could cause problems–I would not be so quick to dismiss the possible bias from overreporting–at least not too much.

Now, the problem that could arise from the type of weighing that USC folk have done is that, if the support for Obama were overrepresented in the sample to begin with, weighing them down to reflect the overall partisan balance would underrepresent them even more than they should be.  It is in this sense where the USC poll might be overrepresenting the support for Trump–by undersampling, in effect, the latent Democratic voters and, relatively speaking, oversampling Republican voters.  But this is not necessarily a fatal flaw once you have the full data:  if someone does say that he or she voted for Obama in 2012–whether it is true or not–that is a valuable piece of information.  This, in turn, can be used to identify other interesting questions, which, although somewhat indirectly, provide an understanding of what’s going on in this election.  Just what kind of voters say that they voted for Obama but would support Trump, for example?  How does this square with the actual electorate that Obama did have in 2012?  Assuming that their answers are actually honest, what implication does this have for potential “swingability” of a given slice of the electorate?

People do not change their partisan stripes readily–at least, not in today’s environment. Whether it is honest or not, that a respondent should say that they were an Obama supporter is significant, as an indicator of their political inclination–and note that USC constructed its panel before the Trump (and Sanders) phenomenon hit.

In a sense, I may not impressed by this critique by NYT because of my own bias:  I don’t care much for aggregate predictions from any poll, but am deeply interested in tracking how different demographics are moving–how their “swingability” (or “variability” in their choice) evolves over time.  When the variability is high, the predictions become inherently hard to make.  The composition of the electorate will be slightly different this time from 2012, to say the least.  We should want to know how it will be different and how different components of the electorate will behave–will they behave as they did in 2012, or will they do something else?  The way USC has been conducting its polls is potentially much better at gauging the variability, even if it may be poor–perhaps!–in predicting the outcome.

PS.  A better way of describing my view towards use of statistics is that, as long as the data itself is true–that is, not made up out of whole cloth–every analysis reveals some aspect of the real world.  The “prediction” is not so much important as the revelation of the real variability in the data, conditional on the approach.  But, the catch is that, in order to learn what a given approach to data has to teach us, we need to pay attention to how they got to their conclusions, rather than their “conclusions.”  It may be highly improbable that Trump is doing as well as the USC polls suggest that he might be.  If he is indeed not doing so well, what is it about the technique used by the USC team that biases the result?  If he is indeed doing so well–and all others are missing it–what are they catching that others have missed.  And, even if Trump is not doing so well, as long as the data is true and the methodology is sound–and both of these seem to be the case–they are still capturing something about the “truth” that others are not, even if the “prediction” might be off.  At this stage, even if we might be quite sure that USC polls are “inaccurate” as the predictor of the results, they are using a novel technique that, quite frankly, makes a good deal of logical sense.  It is in the deconstruction of what they have done that we will learn, not in trashtalking over whether they got the conclusion right or wrong.

The Role of Economy in Trump’s Support

A recent article in the New York Magazine attempts to debunk the misleading implications claimed by the Washington Post regarding a new study from Gallup, and in turn, adds a few misleading implications of its own.

The study itself fails to draw attention to what makes the Trump voters stand out from other Republican voters.  This is critical since the selection into categories “favor Republicans” and “favor Democrats” is not random:  Republican voters, generally, ARE different from the Democratic voters.  By pooling all the voters and treating support for Trump as a single categorical variable, same as support for any other candidate, the study mistakes characteristics of generic “Republicanness” for predictors for support for Trump.  This is the case for the singular implication drawn by both WaPo and New York Magazine, and, in fact, by the study’s author itself:  that Trump’s supporters are not economically badly off.  But the tables themselves show that they ARE in fact worse off economically, for Republicans, at any rate:  they are better off than supporters for either Clinton or Sanders, but they are significantly worse off than supporters for all other Republicans except Ted Cruz (Table 5–which is confusing since support for Trump is the baseline category.  It would have been far better to use twoway ANOVA with contrasts rather than probit for this analysis.).  On the other hand, they are not necessarily more likely to be self-employed (a generic trait for the average Republican voter).  Trump voters are, surprisingly, LESS likely to be retirees than supporters for either of Democratic candidates,  certainly more likely to be union members–at least compared to other Republicans (although much less than for the Democrats), somewhat younger on average (but with a peculiar distribution) than Clinton’s supporters (but with a larger contingent of older cohort–which shows up when the age squared variable is included–which implies heavier tails:  Trump supporters include both youngish and oldish people in larger proportions than Clinton.).  All in all, there is a LOT more nuance in the numbers presented by the paper than meets the eye, and a better methodological choice (say, employing some of the clever techniques proposed by Andy Gelman) would have been handy.

All in all, the New York Magazine article gets it right:  the support for Trump is a rather nuanced phenomenon and it should be approached as such.  They are, on average, rather similar to the Republican voters, but different in significant aspects–which would be indicative of a heterogeneous Republican faction that draws support from many people who are not typical Republican voters in addition to more customary ones.  In other words, as I’d been labeling it, a Bud Light coalition–who may be drawn to it by a mixture of cheap price, lousy taste, cheapish ads, and other heterogeneous reasons but not necessarily a single unifying reason as the supporters for craft beers might be.  Economic reasons are an important reason for many of them, as must be racism, both hard and soft varieties for different people.  But it is not the conventional left-right scale that defines them and forcing them to the procrustean bed of the received wisdom is a dangerous mistake.

The Brilliance of the Clinton Campaign

Carl Beijer makes a point that, contrary to his intentino, underscores an utter brilliance of the Clinton campaign this time around.

Beijer’s point, in a nutshell, is that, because Trump is so unpopular, Clinton can make an electoral killing by tarring and feathering the entire Republican Party with Trump.  Indeed, this is what a recent WaPo Monkey Cage post contends.  The problem is that there is no evidence that the Republican voters would necessarily vote against Trump down the ballot, even if they might vote for a Gary Johnson with their presidential vote.  They can tell that Trump is not a real Republican and a lot of GOP establishment is trying to make that point. The bigger threat for the Republicans is that their voters may not even turn out at all–but this presents a more complex challenge because, for what it is worth, the demographics that tend to vote Republican–college educated, affluent, female, and white tend to turn out and are most “partisan”–even though they are also precisely the demographics that distrust Trump most.

The electoral characteristics of the voters who distrust Trump most implies the possibility that Beijer misses (but is implicitly addressed by Nate Cohn  at NYT)  If Trump looks sufficiently like a regular Republican, many regular Republican voters might reconsider and vote for Trump as they would any other Republican.  By disassociating Trump and the regular Republican voters, Clinton is recognizing the power of partisanship in vote choice.  Are they right?  Perhaps, perhaps not.  But the patterns noticed by Cohn suggests that the Clintons are more right than the naive leftists who think that the entire Republican Party can be hammered with Trump:  I don’t see Democrats winning House seats in Republican districts in Georgia or Louisiana, any more than the Republicans could capture the House in 1984.  Republicans and Trump are different entities and the former has more electoral life beyond 2016.

PS.  One might describe the Democratic problem as a choice between a high risk strategy and a low risk one.  If the Democrats try to tar all the Republicans through their association with Trump, they could conceivably win across the board, or help Trump consolidate Republican support.  As it were, the Republicans and Trump are doing pretty good job tearing each other down so that extra help from the Democrats probably is not necessary on that department, and, if anything, an external threat, in form of coming from Democratic attacks, could help them unify.  This is dangerous for the Democrats because, if Trump can both make serious inroads among the working class whites and retain regular Republican support, he runs a realistic chance of victory.  But, by trying to concede the Republicans down the ballot some electoral room at the cost of depriving Trump of support from the regular Republicans, Clinton ensures that Trump will be totally defeated.  The role of partisanship in vote choice is such that, as long as Trump is THE Republican nominee, possible victory remains within reach, as long as regular Republican voters remain Republicans.  They will not become “Democrats” any more than Democratic voters started voting for Republican candidates for Congress just because of Walter Mondale–they didn’t.  However tempting the thought of overwhelming victory across the board taking advantage of Trump’s seeming unpopularity might be, it is important to recognize that Trump is unpopular largely because he is unpopular with the Republicans because he’s not Republican enough. Carville and Greenberg, for all their smugness, do know the polls, and these are what polls have been telling us for a long time.

PPS.  The somewhat more succinct version of this argument, with input from one of my friends, is that, in order to trumpify all the Republicans, Clinton campaign has to convince Republican voters that the Republican officeholders are not real Republicans, but crazy alt-right looneytoons.  This is not easy.  Of course, to win them back, Trump has to convince the same voters that, compared to him, Clinton is a tax-and-spend liberal of the old sort.  This, too, is a difficult sell–especially since, in a way, she has very much staked her ground as a Republican Lite.  Peculiarly enough, given where she has established her reputation, trying to tear down Trump as a Republican, rather than attack the Republican Party via Trump makes sense for Clinton.  Somewhat ironically, a more conventional campaign–and expected votes–might have been waged had the matchup been between Sanders and Trump.

Critical Thinking and the Variance, Formulaic Thinking and the Mean.

I might have been completely missing the point, but this article in LA Review of Books had me refining my thinking about means-centric thinking vs. variance-centric thinking.

In effect, the article is bemoaning the decline of critical thinking in all manner of venues in favor of “content,” or as the author terms it, #content.  What counts as #content, in turn, is determined by whatever it is that the audience wants.  Or, in other words, the problem quickly comes to resemble the beauty contest game.  The statistically meaningful consequence of beauty contest games, in turn, where the players try to base their decisions not on what they themselves want but what they think other players want is not that they necessarily misjudge, at least on average, but that they become too stereotyped in their thinking, so to speak.  The distribution of true preferences usually feature substantially larger variance than the distribution of anticipated variances.  Put differently, when the media attempts to deliver the #content that they expect that people want, rather than trust their judgment to come up with something on their own, they not be too different on the average across multiple instances, but their variances certainly would be.  In the context of individual draws, in turn, this has a significant implications.

A relatively simple demonstration is in order.  What is the expected distance between observations drawn randomly from a distribution with a small variance and another distribution with the larger variance?  For the sake of simplicity, let us take a standard normal distribution and another normal distribution with the standard deviation of 10.  The distribution of the difference will be distributed as a normal distribution with mean 0 and the standard deviation of around 10.5, but the mean 0 is only due to the negative differences exactly cancelling out the positive differences.  The actual distances (or squares thereof) are distributed as the chi-squared distribution with two degrees of freedom (sort of–since true chi-squared distribution is the square of a standard normal distribution) arising from squaring a normal distribution of the variance 101 (or the square of 10.5).  Of course, this is practically definitional:  variance = E(x squared) – (mean of x) squared.  So as the difference in variance between the reflected conventional wisdom (what the players think other players want) and what the players really want increases, the actual average gap grows as well, on both sides!

It is also striking that this bears a curious resemblance to the coalition politics of “populism” today.  The talk of “dangerous radicals of left and right” became an epithet, but it captures a certain truism:  “radical” political leaders draw support from the left and the right, even if not exactly in the same proportions.  So the conventional politicians operating on the anticipation of what the electorate gets the mean right, but hideously underestimates the variance, and the gap between the reality and their program is increasing, even as they get the mean with ever precise precision, thanks to the Big Data and associated technologies?  This is an interesting thought….

PS.  An important caveat is that being able to guess the mean with ever increasing precision will not necessarily increase the gap.  The key is simply that the gap, if the true underlying data is sufficiently variable, cannot be made to go away even if you do know the mean precisely.  So the more accurate rendering of the argument is that the marginal gain from a more accurate guessing of the mean is small, when the natural variance is high.  Mathematically, the variance of the differences will decrease of the variance of one of the distributions goes down.

An interesting problem emerges, however, if this is viewed in context of polarization.  One might say that both parties, while reducing the variance, have grown farther from the mean of the popular distribution.  So, whereas, in the old era, the distribution of the differences might have been N(0, a+b) where both a and b were fairly high variances, we now face N(c, a+b’), where b’ < b, but c is significant.  So the mean gap squared (i.e. E(mean squared for the distribution of differences)) is now a+b’+ c^2, while it was simply a+b in the past.  If this gap is to be construed as the extent of “mean representation,” it is not necessarily clear if the present parties are any more “representative” than in the past.

PPS.  Yes, I am assuming independence and model completeness in distributions, which assuredly is not the case–even in the high variance era, things like “popular politics” and “home style” assured that outliers were, in fact, reflecting unobserved variables–which, all practical purposes, meant that the outliers in one distribution were correlated with outliers in the other distribution somehow.  But this goes well beyond a simpleminded model.