I was looking through Kaggle the other day and saw that most of them write these lengthy Python methods instead of using Scikit-Learn or any other tool. ex: Are there any benefits to writing Python code directly without the usage of any libraries? For example, why develop a function for training or testing when you can utilize the hassle-free scikit-learn library?
1 Like
You are heading down the right road! Usually, using trusted libraries that have been tested and proven is the smartest move. But if you’re tinkering around for learning purposes, that’s fantastic too! Just make sure to compare your results with something like scikit-learn to see how you’re doing. Sometimes, though, you might have good reasons to go off the beaten path. Like if you’re dealing with special problems, need to be super efficient, or have to play nice with another system. For instance, if you’ve got to translate something into a SQL user-defined function (UDF), you can’t just breeze in with “import sklearn” most of the time.
Oh, and sometimes you’re stuck in a tight spot where you can’t install new packages for some reason, maybe because of strict IT rules or a pressing deadline. In those situations, you’ve gotta get creative! Our team, for instance, had to make do with a Python 2 Spark cluster for a while, unable to install any new packages thanks to “security reasons”. But hey, we’re slowly making the move to Databricks now.
1 Like
As someone who has worked extensively with Scikit-learn, I’ve found that one of the main reasons some people don’t use it for lengthy Python routines is that it’s primarily designed for machine learning tasks, such as data preprocessing, feature selection, and model training. While it does provide some utility functions for data manipulation, it’s not as comprehensive as other libraries like Pandas or NumPy. Additionally, Scikit-learn is optimized for performance and scalability, which can make it less suitable for tasks that require more flexibility or customization. For example, if I need to perform complex data transformations or handle missing values in a specific way, I might choose to use Pandas or other libraries that are more geared towards data manipulation. However, for tasks that involve machine learning models, Scikit-learn is an excellent choice due to its ease of use, flexibility, and high-performance capabilities.
1 Like
Top Kaggle entries are hyper-optimized, sometimes overly tailored to the specific challenge, making third-party libraries a no-go.
1 Like