The topic of the debate is simple-there are folks that think that AutoML means that there will be a drastic reduction in the number of Data scientists and Data engineers that current exist today. Obviously, those who counter this are arguing that while AutoML may automate many aspects, the world will still need plenty of Data engineers and Data scientists.
What is my stand ?
I side with none but partially side with those that believe AutoML is the future. Confused ? Read on to figure out what I mean by that. So first, let me start with the opinion statement of my stand on the AutoML debate:
AutoML WILL replace most of the task that Data engineers and Data scientists currently do. Whatever tasks that the other side is currently arguing that AutoML will allow Data scientists to focus on (while AutoML will focus on the mundane), will also get automated and bundled in subsequent AutoML products. I see task of Data engineers getting completely consumed by AutoML products along with 50-70% of what majority of Data Scientists currently do.
There will still be demand for quality Data engineers and Data scientists and associated services. (Though not at the current “artificially inflated” level).
Why the demand for genuine Data engineers and Data Scientists will remain for many more years ?
Because in my opinion, both sides of the debate are missing one critical piece.
When they think about AutoML, they mostly see them as external, off the shelf offerings from vendors for consumption by organizations.
But there is no rule that says these organizations can not develop their own AutoML products. Then why is our thinking constrained with a vision that AutoML tools will only be available as off the shelf offering by external vendors. More than an year ago, I suggested that developing an off the shelf AutoML product is the worst market to get into, if you already are not neck deep into developing one. Not only is the market getting crowded, I anticipate that leading organizations, in the near term, will prefer to build their own AutoML product due to an important factor, that, in my opinion, does not make off the shelf AutoML products viable product in the long run. I have shared that reason in the subsequent section.
Yes- Companies can and should build their own, customized AutoML products
That is what I mean when I say you need to build your Digital Supply Chain capabilities as tech platforms. Now how AutoML tools plug into that “Digital Supply Chains as platforms” strategy is not within the scope of this article. Coming back to the topic of AutoML-If you, as a buying organization, are in the market for an AutoML product, here is my unbiased, no BS advise:
The only AutoML product that will work perfectly for you is the one you will build for your own unique requirements.
And here is the reason why.
And this reason is also the reason I mentioned in the previous section that I will touch upon. The reason is, while the factories churning “Data scientists” and Data science “Professionals” spoon feed generalized algorithms and libraries to those who graduate from these programs, and while snake oil vendors offer their “AI enhanced” tools as panacea- no two Algorithms, and their associated data sets, characteristics etc. will be the same, if they are solving two different problems. And even though the problem area may be same, the nature of variables differ for the same problem when you move from one company to another, even in the same Industry.
And because of those aspects, with an off the shelf AutoML tool we get into the same trap that we have been in with legacy Supply Chain solutions- force fitting unique problems in tools that are supposed to then spit out “some” solution- which is then manually refined.
Since each organization is solving a unique problem, have unique datbases and datalakes and associated infrastructure, there is no way an enterprise off the shelf AutoML tools will effectively be able to do what it intends to, without too many manual touch points, thereby defeating the entire purpose of having an AutoML tool. Therefore organizations MUST build their own AutoML tools, which will be an essential block in their “Building Digital Supply Chain as a platform” strategy.
And for this they WILL need Data engineers and Data scientists
And this is why I say that demand for quality Data engineers and Data scientists will remain. Building AutoML platforms, that is agnostic to attrition in your Data science teams, takes a deep level of Data science talent. Also prosper will the market for Data science services, that can help clients build customized AutoML products.
So don’t go out looking for talent to build your Data science teams. Go out looking for talent to build your AutoML platforms. That is the key to building sustainable Advanced Analytics capabilities that will plug nicely into your Digital Supply Chain platform architecture.
Views expressed are my own.