By Nayiri Mullinix
"A key design goal was to make it simple for research communities to glue clusters together without expert help, with clustering middleware of their choosing. Our SGCI engagement gave us web workflow patterns and other UX tools to ease the creation and management of virtual clusters for the collaborations we support."
Research Professor, University of Chicago
Imagine a world where researchers from multiple institutions can easily pool together computing resources in order to foster seamless collaboration. Over the last three years, the team behind a project called Virtual Clusters for Community Computation (VC3) has been working to enable researchers to do just that.
How does VC3 make it easy for researchers to collaborate? Rob Gardner, co-PI on the VC3 project, suggested that we think of it like a potluck—where each guest brings a dish to share. The same applies to VC3 but, instead of a dish, each researcher brings an allocation to share. Say, for example, that you’re a researcher at institution A and are collaborating with researchers at institution B and C. You and your collaborating colleagues have each been granted an allocation by your institution which you can then bring to the VC3 platform and connect together. This creates a common pool that VC3 uses to build a service that gives the collaborating researchers a compute cluster. The resulting research space, created by researchers in a self-service manner without the assistance of systems administrators, is bigger, better, and more powerful because both data and CPU time from each institution is being shared. So, a research potluck!
When VC3 came to SGCI for support, much of the heavy lifting required to build up the back-end had already been done, and the team was searching for guidance on web technologies. The gateway is a Python Flask-based application that incorporates a responsive client-side web design via the Bootstrap library, authenticates users against a variety of ecosystems via integrated GlobusID authentication, and dynamically interacts with the VC3 ecosystem via the VC3 InfoService database to produce a full-service single point of access for all VC3 ecosystem users, regardless of institutional affiliation or geographic location. What they needed help with was establishing best practices for the web platform and integrating with the VC3 service framework API. This included advice on building a modern user interface, user authentication mechanisms that integrate multiple heterogeneous ecosystems, user profile management, project and allocation management, and resource and job monitoring as well as administration capabilities.
The consultation with SGCI gave the VC3 team solid footing as far as technology stack, workflow and processes, solving framework-specific challenges, and assistance in identifying how to proceed with vendor code integration. This resulted in a modern, adaptive, and responsive web application which mitigates many of the obstacles that researchers encounter when trying to leverage HPC, HTC, and cloud-based resources. In addition, the VC3 platform fosters global collaboration and lowers the barrier to entry by providing free (or low cost) resources that would otherwise be inaccessible to small or under-funded projects.
Thus far, VC3 has engaged with three communities to build virtual clusters.
Large Hadron Collider at CERN in Switzerland
South Pole Telescope specifically designed to measure the cosmic microwave background (CMB)
Dark matter searches in Italy
What’s next for VC3? Their team, led by Douglas Thain at the University of Notre Dame, is hoping to grow the project in order to support more communities. Part of this growth would also include extending the platform to new types of middleware and language environments.
Read more about VC3:
VC3: A Virtual Cluster Service for Community Computation