Workshop on Schedulers 2016-09-14

A scheduling workshop was organised by Ash Vadgama on 14 September 2016.

 

The WORKSHOP GOAL was to provide a detailed overview of the current HPC Scheduling Marketplace, so as to understand current scheduler capabilities, current deficiencies, developing risks and new features – thereby providing HPC-SIG members with options for the future.

 

The THEME GOAL was to CAPTURE TOP 5 needs, current deficiencies, list of the Risks and Potential Solutions/Opportunities

 

AGENDA and subsequent notes were:

  • 08:30 AM – ARRIVALS and COFFEE – 30 mins
  • 09:00 AM – WORKSHOP START – INTRODUCTION / GOALS and OUTCOMES – 15 mins + Q and A
    • Led by Ash Vadgama (Workshop lead)
  • 09:30 AM – SHORT TALK – INTRODUCTION / DISCUSSION – 30 mins + Q and A
    • SLURM (Lessons Learnt) – 15 mins, Led by Steven Chapman & Roshan Mathew (University of Bath)
    • Scheduling to the Cloud – 15 mins, Led by Matt Harvey (Imperial College)
  • 10:00 AM – WORKSHOP THEME ONE – Global and Federated Scheduling – 1 hour
    • GOAL – CAPTURE TOP 5 needs, current deficiencies, list of the Risks and Potential Solutions/Opportunities
    • Global Scheduling
      • Across multiple local systems, different schedulers or multiple architectures?
      • Capacity on demand scheduling
      • How do you cope/deal with spanning systems?
      • What if you don’t get the resource you request (or just some of it)?
      • Global/federated scheduling
      • Is data movement important?
      • Why not data location aware scheduling?
      • Why should data sets be treated differently from say needing KNL?
  • 11:00AM – WORKSHOP THEME TWO – Heterogeneous and Mixed Scheduling – 1 hour
    • GOAL – CAPTURE TOP 5 needs, current deficiencies, list of the Risks and Potential Solutions/Opportunities
    • Mixed Scheduling
      • Running jobs across different service providers – scheduling between local systems, cloud resources and back?
    • Compartmentalisation
      • A need to compartmentalise the production and teaching clusters and this changes over time
    • Virtualisation Support
      • The tools and support to be lacking – Vendors are pushing OpenStack (Sadly on talking to them it seems they’re making a lot of assumptions about our environment (i.e. it’s openness) that are simply wrong
    • Containers / Docker (like) support
    • Heterogeneity
      • In our case, we have POWER8, ARMv8 and a few others, not just X86
      • Heterogeneous Scheduling
      • Scheduling jobs which use CPU’s, GPU’s and other accelerators – incl. MPI and OpenMP?
      • Heterogeneous resources
      • Challenges in (and solutions to!) scheduling jobs and access in a single HPC environment.
    • FairShare
      • Managing fairness – what does “fair” mean in HE environments, and scheduling within that context.
      • Is a potential metric just “(wait time)/(run time)” – Is FairShare really used/needed?
  • 12:00 PM – LUNCH BREAK – 1 hour
  • 1:00 PM – WORKSHOP THEME THREE – Services and Data Migration – 1 hour
    • GOAL – CAPTURE TOP 5 needs, current deficiencies, list of the Risks and Potential Solutions/Opportunities
    • Service Migration
      • The challenges of migrating a service between schedulers
    • Data migration
      • Scheduler considerations (e.g. additional “copy time” to add to compute wall clock time) when there is a “copy data to scratch -> compute -> copy results from scratch” workflow and;
      • Scheduler level strategies for automating this workflow (disclaimer: we don’t do this at the moment but we do have a requirement to look into it)
      • Staging data in/out
      • Some of our file systems are more sensitive than others, and when we carve off some nodes – we’re still using very hacky workarounds to get the scheduler to stage files in/out properly.
  • 2:00 PM – WORKSHOP THEME FOUR – Exploring Schedulers and Product Features – 1 hour
    • GOAL – CAPTURE TOP 5 needs, current deficiencies, list of the Risks and Potential Solutions/Opportunities
    • Open verses Proprietary Schedulers
      • Free verses Paid scheduling products – which are better?, more cost effective?, and at what scale?
      • Free versus Paid scheduling products – Perhaps we can run a poll before the workshop order to assess the situation on the ground?
      • what scheduler are you using
      • satisfaction with support (1-5)
      • cost (1-5)
      • past experience (schedulers you migrated from)
      • future plans (the scheduler you want to migrate to)
    • relevant topics of interest (a, b, c + more ordered according to urgency or scored from 1-5)
    • Schedulers – Exploring if there are “better” alternatives
      • If it’s a case of “horses for courses”
      • If different products shine in different use cases.
      • Is there a lot to choose between Moab (& Maui) / SGE(& open-source forks) / LSF(/pbs_pro?) (the behemoths of scheduling world), and likewise between torque/slurm (at the other end)
        GRIDPP/CERN – use of different schedulers? – GridPP/WLCG context: https://twiki.cern.ch/twiki/bin/view/LCG/BatchSystemComparison, http://indico.cern.ch/event/272785
    • Product Specific Features (we’d like to see) SLURM
      • Like most people, we’re seriously considering moving to SLURM, but aren’t convinced it will do what we want yet?
    • Workarounds
      • Having to implement hacky workarounds, I wondered if anyone else is having this problem.
  • 3:00PM – AFTERNOON BREAK – 30 mins
  • 3:30 PM – WORKSHOP THEME FIVE – Advanced Scheduling Requirements – 1 hour
    • GOAL – CAPTURE TOP 5 needs, current deficiencies, list of the Risks and Potential Solutions/Opportunities
    • Energy aware scheduling
      • Using scheduler to minimise environmental footprint of computational research.
      • Exploring energy awareness in applications, and in the HPC scheduler (SLURM)
      • Energy scheduling (from a tooling and scheduling point of view)
      • Advanced Techniques in Scheduling through Power Management.(“aligns with planned HPC-SIG – Data Centre and Energy Workshop”)
      • Scheduling jobs with power as a managed constraint/limiting factor (in terms of quantity or cost); monitoring power at the job level to develop efficiency metrics for application performance modelling or algorithm development.
  • 4:30 PM – WORKSHOP WRAPUP – DISCUSSION ABOUT GOALS, OUTCOMES and NEXT STEP – CONTENT OF HPC-SIG WORKSHOP REPORT – 30 mins
  • 5:00 PM – WORKSHOP CLOSE

Christopher Walker from Queen Mary University of London has compiled a list of schedulers in use by delegate sites at the time of the workshop