Note: The article assumes that readers are familiar with basic terminology of Neural Networks.
0. Introduction- Managing disruptions through technology
The disruptive impact of Coronavirus epidemic on Manufacturing and Supply Chain networks has been one of the most widely discussed topics recently and these discussions have gone beyond the world of Supply Chain professionals.
Among these discussions have also been discussions about how AI could have helped us manage the disruption better. This article leverages one scenario that I recently experimented with, where a algorithm can help a Flexible Manufacturing system devise optimal plans around such disruptions.
We will take the following path in this article to articulate the scenario:
- Overview of simulation in Manufacturing (Particularly Discrete event simulation (DES))
- Simulation in Industry 4.0
- Combining Simulation with DRL
- DRL in Flexible Manufacturing systems
I. Simulation (Discrete event) in Manufacturing
Simulation of a manufacturing facility requires modeling and recreation of the behavior and performance of each individual process and system. In establishing the model, it makes sense to breakdown each process into its discrete parts. This makes it easier to analyze and allows more factors to be considered within the model.
Discrete Event Simulation (DES) software approximates continuous processes into defined, non-continuous events. For example, Discrete Event Simulation software in a vehicle manufacturing facility would model the movement of a car part from Assembly into the Paint Shop as two events i.e. the departure event and the arrival event. The actual movement of the component would be represented only by the time lapse between the two events.
The entire manufacturing facility can be modeled as a sequence of operations being performed on passive entities, e.g. components, as they pass through the processing sequences. Although the components are passive, they have attributes that affect the way they are handled and some of these attributes change as the component advances through the processes.
II. Industry 4.0 and the role of simulation
In an Industry 4.0 application, IT innovations such as Big Data and Cloud Operation make real time data available for Discrete Event Simulation. The dynamic processes in a Smart factory enable operational flexibility that can respond to last-minute scheduling changes, hence Discrete Event Simulation software assists in planning to save time, reduce costs and minimize risk in the overall operation.
When the real system can’t be run over and over with different configurations and settings, Discrete Event Simulation software proves easier than mathematical modeling and provides more realistic results.
It effectively compresses time, allowing simulation of operations that may take days to run to be compressed into just a few seconds. Further, the simulation model can be easily adjusted when the effects of scaling up or down need to be studied.
The resulting information answers fundamental questions about the processes and overall system, for example how long a process takes, how frequently some equipment is used, how often rejects appear, etc. Consequently, it provides data on vital issues such as latency, utilization and bottlenecks for direct improvement in an Industry 4.0 setting.
In this way, Discrete Event Simulation assists with:
- predicting the resulting system performance, over time
- discovering how the various parts of the system interact
- tracking statistics to measure and compare performance.
Although time consuming, the modeling stage requires the involvement of operators and personnel who are intimately familiar with the processes. This imparts an immediate sense of user involvement and ownership that can help in the later stage when implementing findings.
To that end, a realistic simulation also proves to be a much easier and faster tool for testing and understanding performance improvements in the context of the overall system, especially when demonstrating end results to users and decision makers.
In summary, Discrete Event Simulation software can provide competitive advantage when an Industry 4.0 installation needs to be optimized and controlled to manufacture the highest quality products in the shortest time to meet demand and maintain profit margin.
III. Enter Deep Reinforcement Learning (DRL)
Simulation modeling has been in daily practical use for decades in the field of manufacturing. It has a very mature community with a vast body of real-world examples. On the other hand, deep reinforcement learning is a new development in the world of Artificial Intelligence (AI), and still mainly considered a research topic.
Common practice in the simulation community is to take simulation models, run experiments (Optimization, Monte Carlo, parameter variation, etc.) and use the outputs to make better decisions about a model’s real-world counterpart. With this approach, a human is needed to experiment with the simulation model and get information from it.
As mentioned earlier, recent developments in deep reinforcement learning have clearly demonstrated that learning agents (computer algorithms) are also very capable of extracting useful decisions (policies) from simulated systems.
It is high time to combine simulation modeling environments with machine learning, especially as interest moves away from gaming challenges and towards business-oriented objectives.
Within Manufacturing, in my opinion, simulation and machine learning techniques can be combined in the following two areas:
• Modelling, simulation and optimization of production processes and process chains,
• Design, control and reconfiguration of flexible manufacturing systems (FMSs)
III.I What are Flexible Manufacturing Systems ?
A flexible manufacturing system (FMS) is a production method that is designed to easily adapt to changes in the type and quantity of the product being manufactured. Machines and computerized systems can be configured to manufacture a variety of parts and handle changing levels of production.
An FMS design methodology is essentially a combination of design of experiments (DoE) technology, Taguchi method, and knowledge based simulation techniques. The design of new FMSs is not a daily assignment, but their re-design, reconfiguration for a new product, or in case of different disturbances is a very frequent task. The application of simulation Techniques is usually time consuming, which is tolerable in the design phase, but it is hardly acceptable in real manufacturing situations.
As a reasonable solution, I believe that a Deep reinforcement learning solution can be a realistic replacement for a conventional simulation model, specifically for Flexible Manufacturing systems.
III.II A Deep learning experiment in Flexible manufacturing system
During the described investigations three-layer back propagation (BP) neural networks were applied for the FMS. Sufficient number of pattern-target pairs were generated by simulation to cover an appropriate broad combination of design, indicative and noise factors. Examples:
- The number of machine tools and the place of measurement were considered as design factors.
- The speed of the robots, and the AGV have been defined as indicative factors.
- The number of scrap in a batch, the machine set-up and maintenance time, furthermore, the frequency and duration of toolchange were selected as noise factors.
In my experiment, the design, indicative and noise factors constituted the 9 elements of input patterns for three-layer networks with the variable number of hidden neurons (9-X-l structure). During learning 104 simulated values were used as targets. The best estimation results for test patterns were reached by the network of 9-5-1 structure (7.3% maximum, and 3,2% average relative error). These results projected success in applying neural networks trained by simulation results for throughput time estimation.
Essentially, the processing efficacy and accuracy of using DRL in this scenario highlighted that it was a much better alternative for real time or rapid fire FMS planning scenarios.
The scenario for the experiment described above was intentionally designed to be simple to experiment and validate. When doing such experiments yourself, you can replace the machine learning policy with superior heuristics and human tailored algorithms. However, the beauty is that no human assistant is involved in the learning process—that is to say, the AI learns a meaningful policy on its own, based on its interaction with the simulation model. If a more realistic case were setup and effectively trained on several adjacent intersections, for example, the learning agent would start to show its true superiority over human curated algorithms.