Is scikit-learn regularly and widely used in production environments?

Hot Discussions Forum
1

The presenter at a meetup I recently attended (an engineer/DS + Ph.D. from a big tech business) said that scikit-learn isn’t usually used in production. To what extent is that statement true? I presume he was talking about larger-scale, “big data” in this context.

1 Like
2

Scikit-learn in Production:

Widely Used:

  • Popularity and Establishment: Scikit-learn is a well-established library widely used in production environments by numerous companies, including large tech enterprises.

Strengths:

  • Versatility and Usability: It offers a broad range of algorithms, is user-friendly, has excellent documentation, and benefits from active community support. These attributes make it suitable for many production use cases.

Limitations for Big Data:

Scalability:

  • Moderate Dataset Handling: While Scikit-learn manages moderate datasets effectively, it may not be the best choice for extremely large datasets (big data).

Performance:

  • Performance Constraints: With very large datasets, training and prediction times may become slow, affecting overall performance.

Suitable Use Cases:

Smaller to Medium Datasets:

  • Optimal for Common Applications: Scikit-learn excels with smaller to medium-sized datasets, which are typical in many real-world scenarios.

Prototyping and Initial Deployments:

  • Ease of Use for Early Stages: Its ease of use makes Scikit-learn ideal for prototyping machine learning models and initial deployments before scaling up.

Alternatives for Big Data:

Scalable Frameworks:

  • Better Options for Big Data: Frameworks like TensorFlow, PyTorch, or Spark MLlib offer superior scalability and performance for big data applications.

Cloud-Based Solutions:

  • Managed Services: Cloud platforms such as Google Cloud AI Platform and Amazon SageMaker provide managed services for deploying machine learning models at scale.

Contextual Validity:

Large Tech Companies:

  • Custom Solutions: Large tech companies may have the resources to develop custom solutions or use specialized frameworks for big data tasks, potentially reducing their reliance on Scikit-learn.

Smaller Companies and Startups:

  • Valuable Tool: For many smaller companies and startups, Scikit-learn remains a valuable tool due to its ease of use and effectiveness in solving various machine learning problems, even in production environments.

Conclusion:

  • Relevance in Production: Scikit-learn remains a relevant and valuable tool in production, particularly for companies that do not exclusively deal with big data. Its strengths in usability, versatility, and community support continue to make it a go-to choice for many machine learning applications.
2 Likes
3

It depends on what you need to do. If a basic sklearn model can solve the problem, then I don’t see why you couldn’t use it in real-world applications.

1 Like
4

Indeed, sci-kit-learn or any other small Python object similar to it can be used for production.

Simply increase the workforce to handle production requests for the API. It is possible to duplicate the sci-kit-learn object among workers in the function for batch processing, such as Dask and PySpark. For minibatch real-time processing, such as PyFlink, it is also possible to duplicate the sci-kit-learn object among UDF workers. Well, perhaps you have your own Hadoop pipeline. In that case, you can use Hadoop streaming to call Scikit-learn objects, and YARN will carry them out.

1 Like