Cloudian has announced an open-source software contribution that fuses PyTorch with local Cloudian HyperStore S3-compatible storage solutions.
This pioneering development is set to accelerate Machine Learning workflows by allowing PyTorch users on AWS Hybrid Edge solutions to directly access local data lakes running on Cloudian storage.
This allows data scientists and AI developers to run ML on data resident in local Cloudian S3-compatible object storage, without the need to move and stage the data into another system.
The ML tasks can also run on local compute resources such as AWS Outposts and Local Zones.
By bridging the gap between S3-compatible object storage systems and ML compute platforms, Cloudian is reducing the need for data migration as part of a ML workflows – with data able to be analysed at the source.
This open-source contribution bridges the gap between distributed S3-compatible object storage systems and ML compute platform, eliminating the dependency on a dedicated parallel file system for Machine Learning workflows.
By enabling direct access to a cost-effective, scalable data repository, Cloudian simplifies the ML process, reducing both complexity and costs associated with data analysis.
The key benefits of this development are:
- Data Sovereignty
- Simplified Workflow
- Seamless Integration
- Local Performance
“With this contribution, we offer the ML community a tool that integrates two of their most important needs: the computational power of PyTorch and fast, flexible access to training data on S3-compatible storage,” a Cloudian spokesperson said.
