by Jacob Bengtson Posted in Technical | May 03, 2024 4 min read One of the worst-kept secrets among data scientists and AI engineers is that no one starts a new project from scratch.In the age of information there are thousands of examples available when starting a new project.As a result, data scientists will often begin a project by developing an understanding of the data and the problem space and will then go out and find an example that is closest to what they are trying to accomplish.
This is a standard practice, but it has some key drawbacks that don’t always get discussed.This includes: There is no guarantee that the code you find is using best practices The credentials of a given author are often vague The environment may not be compatible Security and legal risks With these issues in mind, Cloudera is thrilled to announce the release of Accelerators for ML Projects (AMPs).AMPs are fully built, end-to-end solutions that provide data scientists with a ready-to-go MVP for various AI use cases, significantly reducing development time.
With a single click, AMPs build, deploy, and set up continuous monitoring of enterprise-ready machine learning (ML) applications. Each AMP is a prototype that encapsulates industry-leading practices for tackling complex ML challenges.The workflow—from data ingestion and model training to model deployment—is meticulously defined within a YAML configuration file.This allows for seamless transitions, whether you’re running examples locally or deploying processes automatically in Cloudera Machine Learning. Best of all, every AMP is fully open source.
Even though they are easiest to deploy in Cloudera Machine Learning, each project provides a README with instructions on how to deploy in any environment—another reminder that Cloudera will always be committed to the open source community.Cloudera’s AMP catalog provides three different types of AMPs for you to choose from.(1) AMPs built with Cloudera engineering, (2) AMPs from HuggingFace Spaces, and (3) AMPs built by community contributors.
Now, let’s dive into these 3 unique types of AMPs and how they can be used.Cloudera Engineering AMPs AMPs built by Cloudera engineering provide the largest number of examples to choose from.These AMPs are built and supported by research teams that focus on the latest and greatest in AI and ML.
They go through a rigorous testing and review process to guarantee that they provide the highest quality reference projects for our enterprise customers to choose from.These AMPs are also continuously reviewed and updated to maintain compatibility with new versions of Python and the various libraries they leverage. One of our most popular AMPs in this catalog is the LLM Chatbot Augmented with Enterprise Data.This project demonstrates how to use the popular retrieval augmented generation (RAG) architecture to add enterprise context to the responses of a locally hosted large language model (LLM) using a hosted Milvus instance as a vector store.
This is a great starting point for enterprises looking to leverage their proprietary data for chatbot applications without the risk of exposing that data.HuggingFace Spaces AMPs HuggingFace Spaces are very similar to AMPs, and as HuggingFace is one of the key members of Cloduera’s AI partnership ecosystem, it only made sense to integrate them directly into the AMP catalog.Like AMPs, Spaces are ML demo applications that are self-contained and instantly ready to deliver value upon deployment.
HuggingFace has built an unmatched community of the best and brightest data scientists, and Spaces are where this community shares its best projects.With a staggering 180,000+ projects to draw from, this integration gives Cloudera customers streamlined access to an unparalleled array of projects to choose from. Community AMPs The strength of Cloudera doesn’t end with its engineering staff.Our strength is our community, from solutions engineers to professional services employees embedded in the world’s leading technical organizations to the practitioners who use Cloudera to solve real-world problems every day.
Our community AMP catalog is where anyone can contribute best-in-class solutions to an open-source repository of meaningful projects.This catalog is where we add standout submissions from Cloudera’s global hackathon events.Most recently, we hosted a Climate and Sustainability Hackathon in partnership with AMD.
With over 2,000 participants from across the world, the hackathon invited the brightest minds to contribute solutions that would help combat the effects of climate change.Get Started with Accelerators for ML Projects Today Don’t just take our word for it, try it yourself.We’re offering a free five-day trial for Cloudera on public cloud.
In this trial environment, users have the ability to launch AMPs from our entire catalog.
Find out how AMPs can accelerate your AI use cases, delivering your AI MVP with a single click!
Jacob Bengtson
More by this author
Editor's Choice
Business
Generative AI for the Enterprise
Technical
Building Trust in Public Sector AI Starts with Trusting Your Data
