A scheduling workshop was organised by Ash Vadgama on 14 September 2016.
The WORKSHOP GOAL was to provide a detailed overview of the current HPC Scheduling Marketplace, so as to understand current scheduler capabilities, current deficiencies, developing risks and new features – thereby providing HPC-SIG members with options for the future.
The THEME GOAL was to CAPTURE TOP 5 needs, current deficiencies, list of the Risks and Potential Solutions/Opportunities
AGENDA and subsequent notes were:
- 08:30 AM – ARRIVALS and COFFEE – 30 mins
- 09:00 AM – WORKSHOP START – INTRODUCTION / GOALS and OUTCOMES – 15 mins + Q and A
- Led by Ash Vadgama (Workshop lead)
- 09:30 AM – SHORT TALK – INTRODUCTION / DISCUSSION – 30 mins + Q and A
- SLURM (Lessons Learnt) – 15 mins, Led by Steven Chapman & Roshan Mathew (University of Bath)
- Scheduling to the Cloud – 15 mins, Led by Matt Harvey (Imperial College)
- 10:00 AM – WORKSHOP THEME ONE – Global and Federated Scheduling – 1 hour
- GOAL – CAPTURE TOP 5 needs, current deficiencies, list of the Risks and Potential Solutions/Opportunities
- Global Scheduling
- Across multiple local systems, different schedulers or multiple architectures?
- Capacity on demand scheduling
- How do you cope/deal with spanning systems?
- What if you don’t get the resource you request (or just some of it)?
- Global/federated scheduling
- Is data movement important?
- Why not data location aware scheduling?
- Why should data sets be treated differently from say needing KNL?
- 11:00AM – WORKSHOP THEME TWO – Heterogeneous and Mixed Scheduling – 1 hour
- GOAL – CAPTURE TOP 5 needs, current deficiencies, list of the Risks and Potential Solutions/Opportunities
- Mixed Scheduling
- Running jobs across different service providers – scheduling between local systems, cloud resources and back?
- Compartmentalisation
- A need to compartmentalise the production and teaching clusters and this changes over time
- Virtualisation Support
- The tools and support to be lacking – Vendors are pushing OpenStack (Sadly on talking to them it seems they’re making a lot of assumptions about our environment (i.e. it’s openness) that are simply wrong
- Containers / Docker (like) support
- Heterogeneity
- In our case, we have POWER8, ARMv8 and a few others, not just X86
- Heterogeneous Scheduling
- Scheduling jobs which use CPU’s, GPU’s and other accelerators – incl. MPI and OpenMP?
- Heterogeneous resources
- Challenges in (and solutions to!) scheduling jobs and access in a single HPC environment.
- FairShare
- Managing fairness – what does “fair” mean in HE environments, and scheduling within that context.
- Is a potential metric just “(wait time)/(run time)” – Is FairShare really used/needed?
- 12:00 PM – LUNCH BREAK – 1 hour
- 1:00 PM – WORKSHOP THEME THREE – Services and Data Migration – 1 hour
- GOAL – CAPTURE TOP 5 needs, current deficiencies, list of the Risks and Potential Solutions/Opportunities
- Service Migration
- The challenges of migrating a service between schedulers
- Data migration
- Scheduler considerations (e.g. additional “copy time” to add to compute wall clock time) when there is a “copy data to scratch -> compute -> copy results from scratch” workflow and;
- Scheduler level strategies for automating this workflow (disclaimer: we don’t do this at the moment but we do have a requirement to look into it)
- Staging data in/out
- Some of our file systems are more sensitive than others, and when we carve off some nodes – we’re still using very hacky workarounds to get the scheduler to stage files in/out properly.
- 2:00 PM – WORKSHOP THEME FOUR – Exploring Schedulers and Product Features – 1 hour
- GOAL – CAPTURE TOP 5 needs, current deficiencies, list of the Risks and Potential Solutions/Opportunities
- Open verses Proprietary Schedulers
- Free verses Paid scheduling products – which are better?, more cost effective?, and at what scale?
- Free versus Paid scheduling products – Perhaps we can run a poll before the workshop order to assess the situation on the ground?
- what scheduler are you using
- satisfaction with support (1-5)
- cost (1-5)
- past experience (schedulers you migrated from)
- future plans (the scheduler you want to migrate to)
- relevant topics of interest (a, b, c + more ordered according to urgency or scored from 1-5)
- Schedulers – Exploring if there are “better” alternatives
- If it’s a case of “horses for courses”
- If different products shine in different use cases.
- Is there a lot to choose between Moab (& Maui) / SGE(& open-source forks) / LSF(/pbs_pro?) (the behemoths of scheduling world), and likewise between torque/slurm (at the other end)
GRIDPP/CERN – use of different schedulers? – GridPP/WLCG context: https://twiki.cern.ch/twiki/bin/view/LCG/BatchSystemComparison, http://indico.cern.ch/event/272785
- Product Specific Features (we’d like to see) SLURM
- Like most people, we’re seriously considering moving to SLURM, but aren’t convinced it will do what we want yet?
- Workarounds
- Having to implement hacky workarounds, I wondered if anyone else is having this problem.
- 3:00PM – AFTERNOON BREAK – 30 mins
- 3:30 PM – WORKSHOP THEME FIVE – Advanced Scheduling Requirements – 1 hour
- GOAL – CAPTURE TOP 5 needs, current deficiencies, list of the Risks and Potential Solutions/Opportunities
- Energy aware scheduling
- Using scheduler to minimise environmental footprint of computational research.
- Exploring energy awareness in applications, and in the HPC scheduler (SLURM)
- Energy scheduling (from a tooling and scheduling point of view)
- Advanced Techniques in Scheduling through Power Management.(“aligns with planned HPC-SIG – Data Centre and Energy Workshop”)
- Scheduling jobs with power as a managed constraint/limiting factor (in terms of quantity or cost); monitoring power at the job level to develop efficiency metrics for application performance modelling or algorithm development.
- 4:30 PM – WORKSHOP WRAPUP – DISCUSSION ABOUT GOALS, OUTCOMES and NEXT STEP – CONTENT OF HPC-SIG WORKSHOP REPORT – 30 mins
- 5:00 PM – WORKSHOP CLOSE
Christopher Walker from Queen Mary University of London has compiled a list of schedulers in use by delegate sites at the time of the workshop