How Analytical Models Failed Clinton

Her campaign was so confident in its data that it opted not to do tracking polls in states that decided the election.

By Charlie Cook

The Novem­ber elec­tions pit­ted Demo­crats against Re­pub­lic­ans, con­ser­vat­ives against lib­er­als, Trump-style pop­u­lists and tea parti­ers against the es­tab­lish­ment and con­ven­tion­al politi­cians. An­oth­er con­test, fol­lowed mainly by polit­ic­al afi­cion­ados, matched tra­di­tion­al poll­sters against newly fash­ion­able ana­lyt­ics wiz­ards, some of whom—pre­ten­tiously in my opin­ion—called them­selves “data sci­ent­ists.”

It was well known that tra­di­tion­al polling was hav­ing prob­lems. The numb­ing ef­fect of bil­lions of tele­market­ing calls and the ad­vent of caller ID and voice mail had re­duced re­sponse rates (the per­cent­age of com­pleted in­ter­views for every hun­dred at­tempts) from the 40s a couple of dec­ades ago to the high single di­gits. As they struggled to get truly rep­res­ent­at­ive samples, poll­sters “weighted” their data more than ever be­fore, mak­ing as­sump­tions of what the elect­or­ate would look like on elec­tion days that were weeks, months, or even a year or more away.

Prob­lems with tra­di­tion­al, live-tele­phone polling led to ex­per­i­ment­a­tion and more re­cently a grow­ing ac­cept­ance of new meth­ods like In­ter­act­ive Voice Re­sponse, pop­ularly known as robo-polls, and on-line polling. Each new meth­od brings both good and bad at­trib­utes. As a tra­di­tion­al­ist, I see the new tech­niques as bad ideas whose time is re­gret­tably com­ing.

The oth­er trend is “ana­lyt­ics,” which in­cor­por­ates in­form­a­tion from a vari­ety of sources—Census Bur­eau stud­ies, com­mer­cially avail­able mar­ket data com­bined with past elec­tion res­ults, and con­clu­sions gleaned from polling, voter can­vassing, and eco­nom­ic meas­ures such as the un­em­ploy­ment rate. This “big data” en­able cam­paigns to mod­el the an­ti­cip­ated elect­or­ate, identi­fy voters most likely to be sym­path­et­ic to their can­did­ates, and shape their mes­sages ac­cord­ingly.

The roots of cam­paign ana­lyt­ics go back to the 1970’s when Demo­crat­ic cam­paign con­sult­ant Matt Reese and Re­pub­lic­an con­sult­ant Ed­die Mahe pro­moted a new tech­no­logy branded Clar­itas, a geo-demo­graph­ic tar­get­ing sys­tem centered on life­styles and neigh­bor­hoods based on a mar­ket-seg­ment­a­tion plat­form de­veloped by com­puter sci­ent­ist Jonath­an Rob­bin (Clar­itas is now owned by Nielsen). It was an idea ahead of its time, too ex­pens­ive for most cam­paigns, and it even­tu­ally left the polit­ic­al theat­er al­to­geth­er.

In 2004 the Howard Dean, George W. Bush-Dick Cheney, and John Kerry-John Ed­wards pres­id­en­tial cam­paigns ad­vanced the uses of data to con­tact voters, but it was the 2008 cam­paign of Barack Obama that took ana­lyt­ics to a whole new level. The in­fatu­ation with ana­lyt­ics after Obama’s reelec­tion in 2012 promp­ted some of his op­er­at­ives to say they didn’t need tra­di­tion­al polling any­more.

When Hil­lary Clin­ton began put­ting to­geth­er her 2016 cam­paign, she brought on board many Obama vet­er­ans, go­ing all in for the new tech­no­logy. Don­ald Trump’s gen­er­al-elec­tion cam­paign also em­ployed ana­lyt­ics, though how soph­ist­ic­ated and im­port­ant it was in his vic­tory is a mat­ter of con­sid­er­able de­bate. House and Sen­ate cam­paign com­mit­tees and su­per-PACs also used ana­lyt­ics to vary­ing de­grees.

The re­li­ance, or per­haps over­re­li­ance on ana­lyt­ics, may be one of the factors con­trib­ut­ing to Clin­ton’s sur­prise de­feat. The Clin­ton team was so con­fid­ent in its ana­lyt­ic­al mod­els that it op­ted not to con­duct track­ing polls in a num­ber of states dur­ing the last month of the cam­paign. As a con­sequence, de­teri­or­at­ing sup­port in states such as Michigan and Wis­con­sin fell be­low the radar screen, slip­page that that tra­di­tion­al track­ing polls would have cer­tainly caught.

Ac­cord­ing to Kantar Me­dia/CMAG data, the Clin­ton cam­paign did not go on the air with tele­vi­sion ads in Wis­con­sin un­til the weeks of Oct. 25 and Nov. 1, spend­ing in the end just $2.6 mil­lion. Su­per PACs back­ing Clin­ton didn’t air ads in Wis­con­sin un­til the last week of the cam­paign. In Michigan, aside from a tiny $16,000 buy by the cam­paign and a party com­mit­tee the week of Oct. 25, the Clin­ton cam­paign and its al­lied groups didn’t con­duct a con­cer­ted ad­vert­ising ef­fort un­til a week be­fore the elec­tion.

In fact, the Clin­ton cam­paign spent more money on tele­vi­sion ad­vert­ising in Ari­zona, Geor­gia, and the Omaha, Neb­raska mar­kets than in Michigan and Wis­con­sin com­bined. It was Michigan and Wis­con­sin, along with Pennsylvania (the Clin­ton cam­paign and al­lied groups did spend $42 mil­lion on tele­vi­sion in the Key­stone State), that ef­fect­ively cost Demo­crats the pres­id­ency.

In the end, the na­tion­al polls fared bet­ter than com­monly thought. The Real­Clear­Polit­ics av­er­age of na­tion­al polls showed Clin­ton ahead by 3.2 per­cent­age points go­ing in­to Elec­tion Day, and the fi­nal ABC News/Wash­ing­ton Post, CBS News, NBC News/Wall Street Journ­al, and Fox News polls each had Clin­ton ahead by 4 points (the last CNN na­tion­al poll was taken two weeks be­fore the elec­tion and had Clin­ton ahead by 5 points). She ended up win­ning the na­tion­al pop­u­lar vote by 2.1 per­cent­age points, 48.2 to 46.1. Thus the RCP av­er­age was off by 1.1 per­cent­age points, the net­work polls were off by 1.9 per­cent­age points. They were off by far more in 2012, but nobody no­ticed be­cause the pop­u­lar vote and Elect­or­al Col­lege tally went the same dir­ec­tion. If one buys the ar­gu­ment that the race changed con­sid­er­ably in the last week, for whatever reas­on, then some of these polls may not have been off by much if at all.

Like so many oth­er as­pects of this elec­tion, a lot of small misses ad­ded up to one gi­ant er­ror on the out­come of the elec­tion. In 54 out of our 58 pres­id­en­tial elec­tions, the win­ner of the pop­u­lar vote also pre­vailed in the elect­or­al vote. A good rule of thumb is that if a can­did­ate wins the pop­u­lar vote by at least 2 per­cent­age points, he or she will al­most cer­tainly cap­ture the Elect­or­al Col­lege. So in an elec­tion when one can­did­ate is thought to have a com­fort­able lead of more than 2 per­cent­age points, there is a reas­on­able ex­pect­a­tion that the elect­or­al vote will go in the same dir­ec­tion. But if the fi­nal res­ult is hov­er­ing at the 2-point threshold, that’s a wrinkle that can cre­ate an un­ex­pec­ted out­come, as the Clin­ton team learned to its dis­may.

It was the in­di­vidu­al state polling that badly missed the mark. In Wis­con­sin, Clin­ton led in each of the 32 pub­lic polls from mid-Au­gust on. The fi­nal Mar­quette Uni­versity Law School, gen­er­ally con­sidered to be the most re­spec­ted in the state, had the Demo­crat up by 6 points. She lost by eight-tenths of a point.

In Pennsylvania, Clin­ton led in 37 out of 38 polls be­gin­ning in early Au­gust. CNN’s last poll had Clin­ton up by 4 points, the fi­nal Quin­nipi­ac poll had her up by 5 points, and the Real­Clear­Polit­ics av­er­age had her up by 1.9 per­cent­age points. She lost by eight-tenths of a point.

In Michigan, Clin­ton was ahead in 25 out of 26 polls taken from the be­gin­ning of Au­gust on. The De­troit Free Press’s last poll had her up by four points, and the Real­Clear­Polit­ics av­er­age had her up by 3.6 points. She lost by two-tenths of a point.

It’s worth not­ing that state polls con­duc­ted by news or­gan­iz­a­tions and uni­versit­ies vary enorm­ously in qual­ity and soph­ist­ic­a­tion. Few state-based news or­gan­iz­a­tions spend the kind of money on polling that many once did. Much of the state-level polling is of a dime-store qual­ity, con­duc­ted by polling firms that are even un­fa­mil­i­ar to polit­ic­al pros.

Ex­per­i­enced journ­al­ists might ar­gue that the over­re­li­ance by re­port­ers on both polls and ana­lyt­ics has led to a de­crease in shoe-leath­er, on-the-ground re­port­ing that might have picked up move­ments in the elect­or­ate that the polls missed. As the Michigan res­ults came in on elec­tion night, I vividly re­called that two con­gress­men from Michigan—one a Demo­crat, the oth­er a Re­pub­lic­an—had been warn­ing me for months that Michigan was more com­pet­it­ive than pub­licly thought. I wished I had listened.

The ana­lyt­ic­al mod­els for both sides poin­ted to a Clin­ton vic­tory, al­beit not a run­away. The Clin­ton cam­paign and su­per PACs had sev­er­al of the most highly re­garded polling firms in the Demo­crat­ic Party, yet in the places that ended up mat­ter­ing, very little if any polling was done. So while 2016 wasn’t a vic­tory for tra­di­tion­al polling, it cer­tainly took a lot of the luster from ana­lyt­ics. In the end, big data mattered very little.

CLA­RI­FIC­A­TION: Ac­cord­ing to Kantar Me­dia/CMAG, a firm that mon­it­ors polit­ic­al ad­vert­ising, the Clin­ton cam­paign’s ad­vert­ising star­ted the week of Nov. 1 in Michigan and Oct. 25 in Wis­con­sin. The cam­paign also made a $70 mil­lion na­tion­al ad buy, $59 mil­lion of which would have been pri­or to Oct. 25, and some of that would have gone in­to Michigan and Wis­con­sin. The cam­paign also had field or­gan­iz­a­tions in both states.

December 26, 2016 National Journal