Wednesday, September 8, 2021

Heuristic input to Predictive Analytics

Cover Illustration for Inventing the Future with Haiku: Whistler P2P
computer painting
©2016 Charlene Brown

Because Artificial Intelligence (AI) has the ability to process ‘big data’ it can be applied to huge problems involving complex systems.  However, in order to be reliable in forecasting the future, it needs to incorporate the intuitive aspect of human intelligence.  

We need to find ways to build common sense into artificial intelligence.

It is possible that properly coded ­algorithms might eventually enable a computer to execute heuristic processes, but it is more likely that heuristic processes can serve as a good first step in data analytics by synthesizing data into a form AI can handle.

Glean as much as you can intuitively before you start quantifying and fitting number-crunching formulas.

I figured this out almost fifty years ago while studying what is now called business analytics.  It was called management science when I got my MBA, and operations research before that.  There was probably a lot less data to mine in the early `70s, but it seemed like a lot at the time. 


1.      The Delphi Technique is a method of arriving at a group opinion or decision by surveying a group of experts (usually in diverse fields) who respond anonymously, then have an opportunity to reassess their answers after seeing the aggregated response. It is especially useful when there is no true or knowable answer, such as in policy decision-making, or long-range forecasting.

·         There are apps that make political forecasts by using AI to comb through Twitter – sort of like a really big Delphi study without having Delphi participants’ opportunity to reevaluate their input.

·         The World Economic Forum Global Risks Analysis, “Visualized: A Global Risk Assessment of 2021and Beyond describes a rigorous method of quantifying expert opinion that sounds similar to a Delphi study, except input is not anonymous. See box below.


2.      Debating at a Policy Workshop: Policy resolutions were raised at the Liberal National Convention, held April 8 – 10 on Zoom. Resolutions had originated with various Commissions and Provincial branches, and were presented and workshopped on the second day. Several dozen were put forward for debate and voting on the final day. 

There were over 6000 of us attending the convention (virtually) and, for each resolution put forward, everyone had a chance to request debate (four debaters were selected and debates were conducted only if there were at least 50 requests – a necessarily arbitrary process) and then vote on whether or not to advance the resolution to the Election Platform Committee. That Committee finalized the Liberal Platform for the September 20 Canadian General Election.

 3.      Visualization: Data visualization, which can distill large data sets into visual graphics, can make it easier to understand complex relationships.  However, as determined in Visualizing Unforeseen Results, visualizations are seldom 'stand alone' documents.  Annotated visualizations may provide more easily understood explanations than detailed text-only analytics.

4.      Ideation Sessions: By employing ‘design thinking’ which considers input from experts in different fields – marketing, design and  engineering – working together, disruptive innovation ideation sessions can enrich discussions.  These sessions help participants to imagine 'what if?’ disruptions such as black swans, wild cards and events such as tipping points occur.

o   Black swan events: unpredictable, massive impact, highly improbable, – eg. 9/11, collapse of the Soviet Union, Covid-19

o   Wild cards: imaginable, low probability, high impact   The difference between Black Swans and Wild Cards is that Wild Cards are imaginable because they have precedents (ie predictable to a certain extent – temperature increases, Halley's Comet, 2008 financial crisis, religious conflicts, financial unicorns or alicorns).  

o   Tipping points: action of a system which has become unstable – eg. effect on crop yields of temperature increases.

Conclusion: Synthesizing information through soliciting wide opinion, debate, visualization, pattern recognition, trend analysis and extrapolation, in other words, going as far as you can in parsing the problem intuitively (heuristically), increases the likelihood of formulating a solvable optimization – and increases the chance the answer will actually make sense.