I started my journey in the field of food specific enterprise software more than 30 years ago, during the time when PC networks started to take over from the mid-range data systems and IBM Mainframes. I have worked with data ever since, and was a witness on how reporting technologies have changed over time.
Any type of reporting system or AI system has fundamentally three different components:
- A way to ingest data, which we call in the BI world ETL for Extraction, Transform and Load. This just means that you take data from sources, clean them up and format them in a way that they are sound and easy to process.
- A way to process the data, where all the ingested data is evaluated, turning the raw data into meaningful information.
- Finally, a way to surface the data, a way to create output of actionable information on paper, websites, mobile apps but also control impulses and automated decisions which may control your credit approval or an exception on your production line and rejection of a product.
Data Ingestion
30 Years ago, data storage was very expensive. In the design of our software, we had to make careful choices on what we store, how we store it and how long we retain it. We were limited to storing data in structured files and tables and were pretty much limited to numbers and characters.
Today, we are talking ‘Big Data’: We are no longer really limited in storage space. Storage space has become so cheap, that databases today operate in-memory, just use hard drives and SSD’s to harden the information. We are also no longer limited with the type of data we process: voice, images, videos and other audio signals can be processed. No longer do we capture the information, store them away and retrieve them for processing at a later time, data acquisition and ingestion can happen simultaneously, something commonly referred to as ‘streaming data sets’. Today, we integrate with our Line Control package machines directly and evaluate the state as well as performance of the processing equipment in real time and trigger e.g. maintenance malfunctions for the plant maintenance module even before the operator knows that something is wrong.
Data Processing
In the early days, SQL reporting was something for the IT department. The skills required to process information was limited to a very few, very specialized people with special education. This changed drastically with the rise of MS-Excel, the ubiquitous reporting tool found today on pretty much any office machine. Every User was suddenly able to create their own reports. Today, when I walk into a Sales opportunity, I usually get questions from my prospects:
- How can I replace this Excel Spreadsheet with your system?
- How can I export the information from your system to Excel?
This democratization of evaluations led to changes, where we developed SSM Evaluation tools, Report Editors which allowed customers to create their own reports on their business. With the integration of Python 20 years ago, we laid the foundation to enable customers to extend reports in any way they wanted.
Today, we still use Python in our system, and have recently added the Anaconda Package to our installation. Python and R are today the leading programming languages in the field of Machine Learning and AI. Even cloud AI solutions like Google Tensorflow or AzureML can be directly integrated if needed.
In machine learning, it is still the human that is largely in control. Humans choose the algorithms like regressions for forecasting, clustering for finding commonalities or classification for binary predictions. This has now evolved into more unsupervised learning modules, where the human really does not need to know anymore how the algorithms work, and the computer learns just based on rules and goals. One example that made headlines around the world was Google’s AlphaGo, which learned to play games better than humans do, and no human can understand how the computer came to any given decision. So called ‘reinforced learning’ in this case.
In our own AI department, we use Machine Learning today to process streaming video to identify content of boxes or manufacturing errors in packaging.
The difficulties today are more to define the goals of an AI solution than the methods. It is commonly accepted that humans are the benchmark, but we encounter more and more cases in which machines can beat humans in terms of accuracy and precision.
Data Surfacing
In the past, we were pretty much limited to printed characters on reams of paper to find the information. This evolved into graphical data analysis, first on paper, later on computer screens. BI Solutions automated a lot the aspects of reading the presented information more effectively by allowing the end users to slice, dice, filter, highlight in the speed the user can think.
The data-driven culture
As people learned to work with the data, some became obsessed with it. The actionable information and intelligence relied initially still on humans to review and act upon. Some of our customers embarked then on a journey, which led them to a data-driven culture. No longer where decisions made based on gut-feeling. When more information was necessary, they started to add more data to their information system: machines where integrated to existing processes, new processes and modules were digitalized to get ever more and better information. This starts an iterative cycle of Ingestion, Processing and Surfacing. Over the years our customers have retained a lot of their historic data, we can fairly easily use and augment these in very robust machine learning applications.
Conclusion
Any business today is at some stage of this journey, but they all have in common the need and desire for better and more actionable information. At the point where the actionable information is more accurate and precise than that of humans, machines can take instant control of processes and make automated decisions. Data volumes and datatypes are no longer limits to what can be done, the limit is human imagination.