Machine learning is one of the hot topics in 2017. Besides, it is expected that machine learning would be part of every developer’s toolbox in near future. It would not just be tool for experts or researchers. However, there are still challenges for machine learning professionals.
Even though, workflows are fully reproducible and new features are easily added to models ideally. In reality, a ML model is hard to re-create and new features cannot be added to the model.
Suppose that you need to add date based sql term (e.g. sysdate – membershipdate) into input features. You would be aware of that that’s not an easy task if you have production experience. In fact, ML models are mostly trained in Python or R, and they are reproduced in production on Java. Building models and training should be handled on production. Additionally, you might evaluate to consume and call scripting languages from high level languages. For instance, NodeJS can interact with the Java, Python shell, R project, and also TensorFlow. This reveals that an interface should be developed between data science and application code.
Moreover, ML related things such as input features, model structure (layer and node size), configurations (epoch, learning rate, momentum, activation function), datasets, success metrics, and trained models (weights) are mostly stored nowhere. Common systems should be consumed to represent and manage machine learning models such as Predictive Model Markup Language and ModelDB. This approach provides to comply with standards. In this way, business manner would systematize.
Finally, data science team roles don’t fall in place yet. Even though, data scientist and data engineer are top of the best jobs based on Glassdoor’s research in 2017, definitions of these roles are still transparent. Data engineer responsibilities are mostly expected from data scientists. Workforce vacancy in data science field might be the reason. You may be familiar with presentations including “we are hiring, too” announcement if you attend any ML summit nowadays.