What Exactly is a Long Tail in Machine Learning ?
Andriy Burkov, a thought leader in the field of Machine Learning, recently posted about Self Driving cars on LinkedIn. You can refer the screenshots below but the summary of his insights were: A completely self driven car, with no driver at all, may be very very far despite all advances in technology. He very eloquently explains what the Long tail is in Machine Learning and why it is very relevant.
Fact is- he is right about this. And his this theory also applies to many other “Sci-Fi” assumptions.
The Problem with the Long Tail in “critical ” applications
Long tail in critical solutions, like the Self Driving car, is something that can’t be ignored since it may end up creating life or death situations. In a car that is driving at 70 mph, we can’t afford to let a Machine make a stupid decision. And it is not practical to aim to cover everything in the long tail as well (by training the algorithm for the long tail).
This applies to the “Lights-out” self managed Supply Chain scenarios frequently predicted by “experts” with limited knowledge of how Machine Learning and Artificial Intelligence actually works.
A Self Managed end to end Supply Chain with no human intervention is practically impossible in many Industries.
If you have actually worked in Supply Chain operations, irrespective of the sub function, you know that unpredictable sh*t happens all the time. And that is why a completely “Lights out” algorithm is extremely challenging to develop. Note that when I say Supply Chain, I mean the end to end Supply Chain. Training an algorithm on every possible sh*t that can happen is practically a mammoth challenge (if not possible) and at some point, the time and money cost of doing that may negate the benefits.
Fully automated Warehouses and Manufacturing plants will be a reality for sure but note that fully automated does not mean fully cognizant.
There is a significant difference between Automated and Smart.
And this difference is the the one that will be the biggest hindrance and the practical, real world reason for not having a completely cognizant, self managed Supply Chain over the next two decades.
This is where Human Machine collaboration becomes much more critical
The Most important Supply Chain organization redesign aspect 15-20 years from now will be :
Where to locate the interface of Human Machine Symbiosis ?
The above question has many aspects embedded within it:
- How can the “long tail” scenarios incorporated into job roles for humans ?
- How will these humans monitor and train algorithms while being available for “Long tail” scenarios ?
- Will the same team do the training as well as managing ” Long Tail” or should they be different teams/individuals ?
- How deep should Data Science teams be embedded in this Human Machine symbiosis ?
- Should there be any effort to reduce the “Long tail” human dependency by continuing to train the algorithm on long tail aspects ?
- If yes, how to determine where to let go?
From a Manufacturing and Warehouse perspective, the best way to visualize human-machine partnerships in full automated plants is to divide each plant into two floors: a fully- automated floor in which the high – volume work of the plant or warehouse is done by machines, and a floor where human engineers and technicians work in support of overall operations.
Views my Own.