Report of 2015 UK National e-Infrastructure Survey of HEIs ... · 16. Regional and national...
Transcript of Report of 2015 UK National e-Infrastructure Survey of HEIs ... · 16. Regional and national...
NeISurveyReport-2015
ProjectDirectorsGroup(PDG)
Reportof2015UKNationale-InfrastructureSurveyofHEIsandResearchInstitutes
MartinHamilton(Jisc/NeIPDG-editor)ClareJenner(UCL/DiRAC/NeIPDG)JackyPallas(UCL/eMedLab/Farr/NeIPDG)AlanReal(Leeds/N8/HPC-SIG)AndrewRichards(Oxford)JeremyYates(UCL/DiRAC/SKA/NeIPDG)
UKNationale-InfrastructureSurveycoordinatedforProjectDirectorsGroupbyJisc
NeiSurvey2015ProjectDirectorsGroup(PDG)
ExecutiveSummary 2
Contents
ExecutiveSummary............................................................................................................................................3
Progressreportonrecommendationsof2014NeIsurvey..................................................................................6
E-Infrastructurecapitalinvestmentsandmajorinitiatives................................................................................11
TheStatusoftheRegionalandNationalHPCProjects.............................................................................................12
Regionalconsortia.....................................................................................................................................................12
TheNationalHPCProjects.........................................................................................................................................13
BootstrappingtheAlanTuringInstitute....................................................................................................................13
WhoaretheUK’snationale-Infrastructureproviders?.....................................................................................14
SustainablefundingfortheNeIEcosystem........................................................................................................16
TheHEIProblem........................................................................................................................................................16
NonHEIProviders.....................................................................................................................................................17
E-Infrastructureasasharedservice...................................................................................................................18
Industry............................................................................................................................................................19
Examplesofindustrycollaborations.........................................................................................................................20
AppendixA-Listofsurveyorsandacknowledgements.....................................................................................22
AppendixB-Whowillreceivethissurvey.........................................................................................................22
AppendixC-Listofrespondents.......................................................................................................................22
AppendixD–Servicemanagement:Thesurveyquestions................................................................................24
AppendixE–Servicemanagement:Summaryofthesurveydata......................................................................26
AppendixF–Servicemanagement:Fullbreakdownofsurveydata..................................................................28
AppendixG–Hardware:Thesurveyquestions.................................................................................................54
AppendixH–Hardware:Summaryofthesurveydata......................................................................................56
AppendixI–Hardware:Fullbreakdownofsurveydata....................................................................................58
AppendixJ–Hardware:Summaryofthesurveydata........................................................................................78
NeiSurvey2015ProjectDirectorsGroup(PDG)
ExecutiveSummary 3
ExecutiveSummaryThisisthedraftreportofthethirdannualUKNationale-Infrastructure(NeI)Survey.ThisisintendedtoinformthemanagementanddevelopmentoftheUK’sNationale-Infrastructureforresearchandinnovation.TheNeIiscomprisedofpubliclyfundedfacilitiesandexpertisethataremadeavailabletoacademicresearchersandforindustrialandthirdsectorengagements.ThereisafulltableoffacilitiesavailableinAppendixJ,includingArcher,theUK’snationalservice,andsystemsprovidedbytheSTFCHartreeCentre.
ThissurveyreportsummarisesthecurrentstateoftheUKNeI,reportingonrecentcapitalinvestments,andintegrationactivitiesdesignedtomaketheNeImoreeffectiveandusercentric.Wemakeanumberofrecommendationsbasedonthefindingsfromthissurvey,whichwillhelptoplacetheNeIonamoresustainablefootingandensurethattheNationale-InfrastructureunderpinsUKresearchandinnovationthatisgloballyleading.
Recommendations:
1. Trainingmaterialsandcoursesshouldbeeasilyavailable.Aservicetohostmaterialsandadvertisecoursesandtoassigncoursestoparticularlevelsofskillandknowledgewouldbeveryhelpfulinmakingsureusersgetappropriatetrainingandareabletoprogresseasilyfromleveltolevel.HEIsandResearchDomainsshouldbeencouragedtosharewhatmaterialsandresourcestheycanwitheachother.Traininghastobeprovidedfortrainers.
2. ATrainingFramework,possiblyonanon-lineMarketplace,needstobecreatedtoallowouruserbasetofindoutandaccesstherighttrainingneededfortheirproject.
3. RCUKshouldasamatterofurgencyprepareabusinessplanforcapitalandoperationalresourcestosupportHPCprovisionintheNeI,indicatingtowhatextentthereductionininternationalcompetitivenessandlossofproductivityduringtheperiod2016-18canbemitigated.ThiswillbenecessaryinordertorealisetheambitionsofinitiativessuchastheTuringInstitute.
4. ThenewlyestablishedRCUKCloudWorkingGroupshoulddevelopastrategytoestablishthescopeandpotentialofCloudservicesforresearchandinnovation.ThisshouldidentifywhichpartsoftheNeIcouldusefullyadopthybridCloudtechnologies,andhowCloudtechnologiescouldassisttheNeIinsharingresourcesbetweenitsdifferentprojects.
5. ToleverageUKinvestmentsinEUandinternationale-InfrastructuressuchastheEuropeanScienceCloud,EGIandEUDAT,RCUKshouldreportonthecurrentlevelofUKparticipationintheseprojectsanddeviseastrategytoensuretheUKbothmaximisesuse,andinfluencesthedesignandmanagement,oftheseinfrastructures.
NeiSurvey2015ProjectDirectorsGroup(PDG)
ExecutiveSummary 4
6. HPC-SIGhasakeyroletoplayinfacilitatingNeIproviders’self-organisationintopartnerships/consortiatobuildcriticalmassofknowledge,achieveoperatingefficienciesandmaximiseproductivity,creatingeconomiesofscalewithoutcompromisingdiversity.
7. AneconomicsustainabilitymodelofthecurrentNeIshouldbeconstructedasamatterofurgencyandasustainabilitystrategydevelopedbythemainparticipantsoftheNeI.Workshopswillbeheldtodevelopthismodelandtodeveloppartnershipsinserviceprovision.ThisisnecessarytoassistwithplanningandtoallowcomparisonofservicecostswithPublicCloudproviders.
8. E-Infrastructureprovidersneedtobegivenconfidencethatthereisalongtermplaniftheyaretoreleasefundsthatwillunderwritestaffrolesbeyondthelifetimeofcurrentprojects,andtodevelopacareerstructureforstaffworkingin“ResearchOperations”(ResOps).
9. RCUKshouldproposecommonresearchandinnovationrelatedmetricsforallNeIproviderstoensureastandardofreportingtoallowtheeffectiveness,efficiencyanddiversityofNeIserviceprovisiontobeassessed.
10. ServiceProvidersshouldbeencouragedtoformpartnershipsandpoolresourcestodelivereconomiesofscaleinprovidingafullrangeofservicesacrossacademic,industryandthirdsectororganisations.
11. Increasetrainingandupdateservicemanagementprotocolsappropriately,sothatthoseusingandmanagingNeIservicesareawareofthesecurityrequirementsoftheirprojectsandtheirsystemsrespectively.Thisshouldbedoneinaproportionatewaythatreflectsdatabeingprocessedanduses/usersofthesystems.
12. ExplorethepotentialofusingJisc’sVATcostsharinggrouptoreducethefrictionofsharingfacilitiessuchase-Infrastructure,andresearchequipmentmoregenerally.
13. TheELCshouldorganisemeetingsbetweenregionalconsortia/partnershipsandNationalProjectswithLEPsandinitiativessuchastheNorthernPowerhouseandMidlandsEnginetoensurethatlocale-Infrastructureneedsareassessed.
14. IUKshouldworkwithregionalconsortia/partnershipsandNationalProjectstodeveloptheCatapultsandworkeffectivelywiththeresearchcommunityandgovernment.ExamplesincludethePrecisionMedicineCatapultwhichwillhaveitsHQinCambridgeandtheMedicalTechnologyCatapultwhichbelocatedclosetotheHartreeCentre.
15. IndustryshouldhaveimprovedvisibilityofNeIresources,e.g.viaaccesstodashboardsandportalsdevelopedbytheNeIforitsownuse.
NeiSurvey2015ProjectDirectorsGroup(PDG)
ExecutiveSummary 5
16. Regionalandnationalfacilitiesshouldhaveanindustrialengagementstrategyincludingmetricsthatpermittheeffectivenessofthestrategytobequantified.
17. LeverageexistingandnewRCUK-fundedbusinessdevelopmentstafftosupporte-Infrastructurerelatedelementsofindustryengagement,workingwithregionalcentresandlocalHEIs.
18. EncourageuseofstrategicfundingsuchasImpactAccelerationAccountstopump-primebusinessdevelopmentactivitieswithinHEIsspecificallyforcomputeanddataresources.
19. ThenewAAAIapplicationSAFE+AssentandthesecureaccessandsecuredatatransportinfrastructureSafeShareshouldbetestedandproductisedinreadinesstoberolledouttotheNeI.Asustainabilitymodelisalsoneededforboththeseservices.
20. End-to-endnetworkconnectivitydiagnosticsshouldbegintobeundertakenbetweenNeIprovidersandusers’localresearchorganisationstoidentifynetworkbottlenecksandprovideresearchorganisationswithadiagnosisreport.
21. CommoncoreapplicationsfortheUKDatae-Infrastructureinordertomoveandselectdatafromdistributeddatasetsshouldbedevelopedandtested.
22. AnRCUKmetadatastrategytobedevelopedtoensurethatmetadataisgeneratedatthepointofdatacreation,andthatstandardmetadataqueryandanalysistoolsareprovidedtoUKresearchers.
NeISurveyReport-2015
ProjectDirectorsGroup(PDG)
Progressreportonrecommendationsof2014NeIsurvey
1.TheconnectionsofNeIServiceproviderstotheSJ6backbonebeevaluatedandifnecessaryupgradedorseparatelinksbeprovided.
2.Internalinvestmentbyinstitutionsisrequiredtoensurethatinternalcampusnetworksremainfitforpurpose.ThisneedstobecommunicatedtoHEIsviatheirPro-VCsforResearch.
JischaverecruitedateamledbyTimChowntodeliveranend-to-endnetworkconnectivitydiagnosisservice.FromtheAutumnof2015thisteamwillworkwithNeIProjectsandHEIstotesttheefficacyoflinksfromlocalinstitutions’servicestotheSJ6network.Thesetestsandasuggestedinvestmentstrategywillbemadetotheparticipatinginstitution.Thisshouldeasedatacongestionwithinthenetworkinfrastructureoperatedbytheinstitutionandincreaseresearchproductivity.
3.AlongtermcapitalplanisrequiredtoensurethefutureproductivityoftheNeI.Thisshouldbeco-ordinatedandcarryonmomentumestablishedtocreateaholistice-Infrastructureeco-systemfortheUK.Beingabletoplanwillgreatlyincreasetheefficacyandefficiencyofoursystemsandreleaseresourcesforaddedvalueactivitiessuchassoftwareengineering.Greatercoordinationhasthepotentialtoestablishdeeperandmorevaluablepartnershipswiththevendorcommunity.
TheUKNeICommunityproducedfourkeyreportstoaddresshowwecreateanintegratedholistice-Infrastructure:
• TheRCUKE-InfrastructureRoadmap,producedbytheRCUKNeIGroup,whichputforwardacoherentstrategytointegrateanddeveloptheUKNationalE-InfrastructuresothatitcandriveforwardthecontinueddevelopmentofagloballycompetitiveresearchbasewithintheUK.Seehttp://www.rcuk.ac.uk/RCUK-prod/assets/documents/documents/RoadmapforELC.pdf
• TheRCUKDataforDiscoveryWorkshopReport,producedbytheRCUKNeIGroup,thatproducedrecommendationsfortheformationofaUKNationalDataforDiscoveryInfrastructure.Seehttp://www.rcuk.ac.uk/RCUK-prod/assets/documents/documents/RCUK%20DataforDiscoveryWorkshopReport.pdf
• CloudComputingforResearchandInnovation,producedbytheNeIProjectDirectorsGroup,whichproducedastrategyformakeuseofcloudtechnologiesandcloudproviderstoenhancee-infrastructureoutcomesin
NeiSurvey2015ProjectDirectorsGroup(PDG)
Progressreportonrecommendationsof2014NeIsurvey 7
theUK.Seehttps://www.scribd.com/doc/273829152/Cloud-Computing-for-Research-and-Innovation
• ImaginingtheUKNationalDatae-Infrastructure,producedbytheNeIProjectDirectorsGroup,whichidentifiedthecoretechnologiesrequiredtopowerafutureUKNationalDataforDiscoveryInfrastructureSeehttps://www.scribd.com/doc/260531862/Imagining-the-UK-National-Data-Infrastructure-
Recommendations
TheRCUKNeIGroup(Chair:Morrell)continuestoco-ordinatee-InfrastructurestrategyamongsttheResearchCouncils,InnovateUK,JiscandtheMetOffice.
InadditiontheRCUKNeIGrouphasfunded(£40k)theNeIProjectDirectorsGroup(Chair:Yates)sothatitcanbetterco-ordinatee-infrastructureintegrationactivities.ThishasallowedthePDGtoorganisemeetingsandworkshopson
• Authentication,AuthorisationandAllocationInfrastructure(AAAI)• CloudTechnologyandCloudProvision• CoreTechnologiesforaNationalDataInfrastructure• DisruptiveHardwareTechnologyanditseffectsonSoftwareneedsanddevelopment
RCUKandPDGmembersmadeinputsinto2016-2021scienceandcapitalplanhttps://www.gov.uk/government/publications/our-plan-for-growth-science-and-innovation
4.JisciswellplacedtocontinuecoordinatingeffortsintheAAAIarea,buildinguponexistingworkwhereverpossible.
JiscannouncedthenewAuthenticationInfrastructure,formerlyknownasProjectMoonshot(seehttps://www.jisc.ac.uk/assent)tomanageaccesstobothwebandnonwebservices.ThiswillformthebasisofasinglesignonservicefortheNeI.
5.DiRAC,GridPPandJiscworktogethertoproduceaprototypethatallowstheseNeIProjectstoshareresources.Thisteststhesingle-sign-oncapabilityoftheproposedcommonAAAIinfrastructure.
Jisc,eMedLab,DiRAC,GridPP,TheUniversityofOxford,TheN8RegionalHPCServiceandtheEdinburghParallelComputingCentreareparticipatingintheSAFE+AssentProjecttoproduceaprototypeAAAIservicethatcanprovidesinglesignonservicesfortheNeI.ThiswillincludeaprojectaroundsecureaccessandfederatingwithprojectsthatareunabletouseAssentsuchasGridPPandElixir.
6.eMedLab,FarrInstitute,ARDC,EBI/ELIXIRandJiscworktogethertoproduceasecureAAAIinfrastructurethatprotectsthesecurityofthesensitivepeopledata.ThisteststhedatasecurityaspectoftheproposedcommonAAAIinfrastructure.
NeiSurvey2015ProjectDirectorsGroup(PDG)
Progressreportonrecommendationsof2014NeIsurvey 8
JischavefundedaprojectSafeShare,£960k,whichprovidesbotha2factorAuthenticationinfrastructureandsecuredatatransport(seehttps://community.jisc.ac.uk/groups/safe-share-project).ThisdoesnotincludetheLifeSciencesprojectELIXIRwhohavetheirownsystem,developedinFinland,butthiswillbeableforfederatewithSafeShare/Assent.
TheSafeShareprojecthasanobviousroleinprovidingsecureaccessdistributeddataanddatatransferavailabletoothersectorsintheeconomysuchasHealthandAdvancedManufacturing.
7.JASMIN,SangerInstitute,GridPP,SKAandDiRACshouldplantoworktogethertoexplorethepracticalitiesofthisapproachandshowthatresourceswithinResearchDomainscanbeconfiguredintoeffectiveandefficientprivatecloudsthatallowresearcherstoruntheirworkflowseasilyonadomainprivatecloudoronresourcesinanotherpartoftheNeI.
GridPP,STFCScientificComputingDivision,SKAandDiRAChaveformedanumbrellaprojectUK-T0todevelopthisapproach.ThisbuildsontheprogressalreadymadebyprojectssuchastheCERNCMSexperiment,theMRCMedicalInformaticsCLIMBandeMedLabprojects,theEBI’sEmbassyCloudandJASMIN2.
SKAhavebegunaprojectwiththeUKsoftwarecompanyCanonical(http://www.canonical.com/)todeveloptheOpenStackCloudoperatingsystemforuseindataintensiveprojectsandapplications;thusmakingOpenStackmoreattunedtotheneedsofUKdataintensiveprojects.
8.Jisctotakethiswork(publiccloud)forward,incollaborationwithmajore-Infrastructureserviceusersandpubliccloudproviders.Howeverthecostsandbenefitsofsuchaccesswillhavetobecarefullymeasuredanditisunlikelytobeasolutionformanyofourproblemsetsinthenext2-3years.PublicCloudProvidersandNeIprovidersshouldexchangeinformationconcerningtheproblemtypesandsizesintheNeI,sothatexpectationsarenotundulyraisedandthatmethodologiesarebuiltupthatalloweffectiveandeconomicuseofPublicClouds.
JischavesurveyedtheCIOsoftheUK’sHEISectortoassessCloudusageingeneralintheHeISectorandfoundthattherewerearangeofissuespreventingmorewidespreadadoptionofcloudincludinglegalandregulatoryaspectsandlackofclarityoverpricingandcostingfor“bogstandard”enterpriseITapplications.See:
https://www.jisc.ac.uk/news/uk-education-divided-in-its-adoption-of-the-cloud-14-jul-2015
JisccontinuetoworkwithPublicCloudprovidersonaccessportalsandegressontotheSJ6backbone.JiscareinvestigatingtheofferingofabrokeringservicetoallowmoreeconomicuseofPublicCloudservices.
ThePDGproducedareportCloudComputingforResearchandInnovation.
RCUKNeIGrouphaveestablishedaCloudWorkingGroup(Chair:Kershaw,RAL),todrivethedevelopmentofprivatehybridcloudsintheNeIandimproveaccesstoprivateclouds
NeiSurvey2015ProjectDirectorsGroup(PDG)
Progressreportonrecommendationsof2014NeIsurvey 9
AnOpenStack(theleadingopensourceCloudTechnology)UserGrouphasbeenformed,supportedbytheUKSMEOCF,toacceleratetheuptakeofCloudtechnologiesbyUKNeIproviders.
9.Thee-InfrastructureLeadershipCouncilisencouragedtoconsiderpotentialapproachessuchasgreaterregionalcollaborationsupportedbyanelementofmatchedfundingfortheHEIs’e-Infrastructureinvestments.ThisshouldbehighlightedtoPro-VCsforResearch.
WewelcometherefreshedcommitmentfortheE-InfrastructureLeadershipCouncilunderthenewadministration.TheirroleinprovidingstrategicadvicetotheNeIcommunityisvaluableparticularlyaswegrowourengagementwithindustry.
ThissurveyreportmakesanumberofrecommendationsonhowtheNeIcanbettersupportIndustryandspecialistareassuchasHealth.
10.ThereisnowtheopportunitytoformallyrecogniseanddefinethecontributionoftheNational“Centres”totheNe-Iandtomakesuretheyareadequatelyfundedtocarryouttheirindividualmissions.
Thisstillneedstobedone.
11.Fundersshouldmakeapplicantsawarethatitispermissibletoapplyforresearchsoftwareengineertimeongrants,andthatitisalsoappropriatetoclasstheseasresearchstaffongrantswheretheworktobecarriedoutinvolvesasignificantresearchanddevelopmentaspect.
Thishasbeendone,althoughitwilltaketimetoworkthroughthesystemasgrantpanelsneedtounderstandthisworkimprovestheproductivityofresearchprojects.
12.Theworktoraisetheprofileoftheresearchsoftwareengineershouldcontinue-inparticulartherole,valueandpotentialcareerpathsshouldbehighlightedintosubmissionstotheELCandHEIs.ThevalueofdedicateddevelopersupportatHEIsshouldalsobehighlightedtoPro-VCsforresearchandDirectorsofResearch,aswellastosuccessfulfundingmodels.
TheEPSRCcallforResearchSoftwareEngineeringFellowshipswasissuedinMay2015,providingsupportofupto£3.7MforResearchSoftwareEngineers.,andanetworkledbytheSoftwareSustainabilityInstitutewhichisopentoallResearchSoftwareEngineers.
Seehttps://www.epsrc.ac.uk/funding/calls/rsefellowships/
13.Trainingmaterialsandcoursesshouldbeeasilyavailable.Aservicetohostmaterialsandadvertisecoursesandtoassigncoursestoparticularlevelsofskillandknowledgewouldbeveryhelpfulinmakingsureusersgetappropriatetrainingandareabletoprogresseasilyfromleveltolevel.HEIsandResearchDomainsshouldbeencouragedtosharewhatmaterialsandresourcestheycanwitheachother.Traininghastobeprovidedfortrainers.
NeiSurvey2015ProjectDirectorsGroup(PDG)
Progressreportonrecommendationsof2014NeIsurvey 10
14.ATrainingFramework,possiblyonanon-lineMarketplace,needstobecreatedtoallowouruserbasetofindoutandaccesstherighttrainingneededfortheirproject.
TheArcherNationalServicehasmadeplacesonitscoursesavailabletothosenotfundedbyEPSRCandNERC,providedresearchersfundedbythoseResearchCouncilsalsoattendthosecourses.ThesecoursesareheldnowheldatmanysitesaroundtheUK.
TheArcherNationalServicehasproduceditsownDrivingTesttocheckandimprovethebasicITskillsofitsnewusers.BothDiRACandArchernowdemandsuchtestsaretaken.
TheSoftwareSustainabilityInstitutecontainstofocusonimprovingtheskillsofthoseenteringtheNeIviatheHEIsector.Theseincludetrain-the-trainertrainingsessions,softwarecarpentryworkshopsandtheprovisionofteachingmaterials.
DiRACandArcherhaveproducedamanycoreprogrammingcourseandaprogrammingcoursefortheIntelXeonPhi,whichwillbemadeavailabletotheNeI.BoththeSKAandSES5haveproducedmanycoreGPUprogrammingcourses.
DiRACandArcherhaveproducedacodeoptimisationcoursethatwillbemadeavailabletotherestoftheNeI.
TheHartreeCentrecontinuestoofferits3weekSummerSchooltotheacademiccommunityandindustry,aswellasafullprogrammeoftrainingeventsaimedatindustry.
SangerandEBIoffertrainingattheircampusatHinxtontoover1500earlycareerresearcherseachyear.
Thepotentialandpracticalityofanonlinetrainingmarketplaceandtrainingresourcessharinginfrastructurehasstilltobeinvestigated.TheBBSRCfundedGOBLET(http://mygoblet.org/)andTeSS(http://elixir-uk.org/training-platform)projectsshouldbeinvestigatedforpotentialtransferabilitytootherdomains.
15.Aprocessisputinplacetomakesuretheon-rampcentresandtheNeIworktogethersothatSMEscanactuallyaccessNeIresources.ThisshouldincludeschemestoinduceSMEstouseNeIinfrastructure.
InnovateUKhavesetupthee-InfrastructureSpecialInterestGrouptoaddressthisissue.MembersofIUKandtheKTNaremembersoftheRCUKNeIGroupandthePDGandmeetregularlywithothermembersofbothGroups.
Seehttps://connect.innovateuk.org/vi/web/high-performance-computing
NeiSurvey2015ProjectDirectorsGroup(PDG)
E-Infrastructurecapitalinvestmentsandmajorinitiatives 11
E-Infrastructurecapitalinvestmentsandmajorinitiatives
The2014SurveydetailedongoinginvestmentsinBigDataAnalyticsProjectsintheareasofHealthInformatics,MedicalInformatics,LifeScienceBioinformatics,AdministrativeDataResearch,AnalysisofPopulationandBusinessData,EnergyEfficientComputing,theSquareKilometreArrayProject,EarthObservation,andDigitalArtsandHumanities.Thisrepresentedinvestmentinexcessof£300mine-Infrastructure.
Manyoftheseprojectshave(oraboutto)comeonlinein2015andcontributetotheUK’sresearchoutputs.
Severalnewinitiativesindatascienceandsimulationhavebeenannounced.ThesestrategicinvestmentsenhancetheUK’sabilityanalyseandmodeldatainkeyeconomicareassuchasHealthandSocietyandWeatherandClimateChange.
Project ResearchOrganisation Amount/£M
CognitiveComputingandDataScience
TheHartreeCentreSTFC(withIBMWatson)
115(BIS)
200(IBM)
AlanTuringInstitute EPSRC,UCL,Edinburgh,Warwick,Oxford,Cambridge
42(BIS)
25(HEIs)
GenomicsEngland NHSandMRC 24
TheMetOffice BIS 97
BigDataandtheInformationEconomy
ESRC 75
TOTAL £378m
£200m(IBM)
NeiSurvey2015ProjectDirectorsGroup(PDG)
TheStatusoftheRegionalandNationalHPCProjects 12
TherehavealsobeenseveralinvestmentsinlocalHEIprovision,notablyatUCLwith£2mbeinganinvestedina185Gflop/ssystem.Southampton,ImperialandOxfordarealsoconsideringsimilarinvestments.
TheStatusoftheRegionalandNationalHPCProjects
RegionalconsortiaAtalevelabovethelocalinstituteresourcesthereexistsalevelofregionalHPCresource.FiveoftheseregionalHPCcentreswerecreatedwithseedfundingbyEPSRCin2011/2012.Thecurrentstatusoftheseregionalcentresvariesandissummarisedbelow.
Regional ConsortiaHEIs Status
SES5 Oxford,Cambridge,Imperial,Southampton,UCL
AllservicesendJuly2015.
LossofmainUKGPUservice.
Imperial,UCL,OxfordandSouthamptonallhaveplanstoreplace/havereplacedtheseservices.
Lossofcollaborationaroundindustrialengagement
N8 Leeds,Manchester,York,Sheffield,Liverpool,Durham,Newcastle,Lancaster
MainN8Servicehardwaresupportendsinearly2016.
HPCMidlands Loughborough,Leicester SupportuntilJune2016foritsHeraHPCserviceandtheyexpecttocontinuetobeoperationaluntilthen.
ARCHIE-WeSt Strathclyde,Glasgow,GlasgowCaledonian,WestofScotlandandStirling
Hardwaresupportuntil31/3/17.
Hardwarealready3.5yearsold.
MidPlus Warwick,Birmingham,Nottingham,QueenMaryUniversityofLondon
MidPlushassupportagreeduntiltheendofMarch2016
NeiSurvey2015ProjectDirectorsGroup(PDG)
TheNationalHPCProjects 13
TheNationalHPCProjects
Threeprojectsreceivedsignificantfundingin2011-2012tosetupbaselineHPCresourcesintheUKtosupportsimulationservicesforacademiaandindustry.
• TheDiRACHPCFacility.ThisFacilitybecameoperationalinAutumn2012andisnowthreeyearsold.Itsupportsresearchintheoreticalparticlephysics,solarandplanetaryscience,astrophysicsandnuclearphysics.ItcontainsoneoftheUK’spetaflopsystemsandclustersdesignedtocarryoutdataintensivemodellingandsimulationtasks.ThisFacilitywillbeuncompetitivefromtheAutumnof2015.
• TheHartreeCentre.ThisFacilitybecameoperationalinAutumn2012andisnowthreeyearsold.ItsupportsinnovativesoftwaredevelopmentforIndustryandprovidessimulationservicesforIndustry.ItcontainsoneoftheUK’spetaflopsystemsandhassystemsdesignedtocarryoutdataintensivecomputing.ThisPetaflopFacilitywillbeuncompetitivefromtheAutumnof2015.
• TheNationalServiceArcher.ThisFacilitybecameoperationalinAutumn2013andisnowtwoyearsold.Itsupportssimulationfortheengineering,physicalsciencesandnaturalenvironmentresearchcommunities.ItcontainsthelargesttheUK’spetaflopsystemsandistocarryoutcomputeintensiveresearch.Thisfacilityhasaplannedfiveyearlifespan,terminatingin2018.
• ThethreeFacilitiescontaindifferenthardwareconfigurationsthataretunedtosolveaparticularclassofproblems.WorkloadsfromeachFacilitywouldnotrunoptimallyonanotherFacilityandwouldinterferewiththecoremissionofeachFacility.
BootstrappingtheAlanTuringInstituteWhilsttheAlanTuringInstitutewasstillintheprocessofbeingsetupastheNeIsurveywasbeingundertaken,werecognisethatwithnocapitalallocationforitsownfacilitiestheATIwillbewhollydependentontheNeIforitsdatascienceandcomputerequirements.WithoutasustainablefundingmodelfortheNeI,projectsliketheTuringInstitutewillstruggletogetofftheground,andmaystruggletoachievetheirobjectives–chiefamongstwhichistoimprovetheproductivityoftheUKeconomy.
RCUKshouldasamatterofurgencyprepareabusinessplanforcapitalandoperationalresourcestosupportHPCprovisionintheNeI,indicatingtowhatextentthereductionininternationalcompetitivenessandlossofproductivityduringtheperiod2016-18canbemitigated.ThiswillbenecessaryinordertorealisetheambitionsofinitiativessuchastheTuringInstitute.
NeiSurvey2015ProjectDirectorsGroup(PDG)
WhoaretheUK’snationale-Infrastructureproviders? 14
WhoaretheUK’snationale-Infrastructureproviders?
TheHEIsremainthemainproviderofdataandcomputeservicestotheNationalE-Infrastructure.37HEIsareprovidingservicesintotheNeI.TheserangefromtheEdinburghParallelComputingService,whichprovidesPetaflopservicestotheNationalHPCserviceandDIRACrespectivelyandpetascaleresearchdataservicestotheUKresearchcommunity,totheUniversityofCranfieldwhichprovidesservicestoengineeringsimulationprojects.
TheseservicessupportbytheHEIsaretheNationalServiceArcher,theResearchDataFacility,DiRAC,GridPPTier2sites,theESRCAdministrativeDataCentreandAdministrativeDataCentres,MRCandCharityfundedMedicalInformatics,theFARRInstitutesHealthInformatics,LifeScienceBioinformatics,theSKA,theRegionalHPCCentres,aswellaslocalprovisionatHEIs.ThesecoverthewholerangeofresearchfundedbytheResearchCouncilsandHEFCE.
JiscprovidesveryhighqualitynetworkservicesbetweenresearchorganisationsviatheJanetnetwork.
TheInternational,CharityandResearchCouncilCentralLaboratoriesprovidespecificlargescaleservicesintheareasofGenomics(SangerInstitute)andBioinformatics(EMBL-EBI),supportforcentralexperimentalfacilities(STFCSCDprovidesservicesforDiamond,GridPPTier1,JASMIN2,ISIS,UKObservatories),SoftwareDataandInnovationservicesforIndustry(theSTFCHartreeCentre).
TheMetOfficeandtheEuropeanCentreforMedium-RangeWeatherForecasts(ECMWF)arethemainproviderofservicestotheweatherforecastingandclimatemodellingcommunities.BoththeseFacilitiesprovidepetascaleservicestotheirusercommunities.
AnewdevelopmenthasbeentheuseofPrivateCloudandPublicCloudservicestoprovideresourcestoNeIusers.Thisiswhollyintheareaofdatafordiscoveryarea.
• GenomicsEnglandhaverentedresourcesfromaUKbasedITprovider• SeveralHEIsandtheMedicalInformaticsprojecteMedLabhaveco-locatedintheInfinityDatacentreat
Slough.Jiscassistedwiththisco-locationproject.AnOpenStackprivatecloudisbeingputinplacetomanagetheseco-locatedservicesinanefficaciousmannertoincreaseresearchoutcomes.
• AnincreasingnumberofHEIbasedprojectsaremakinguseofPublicCloudprovisionandaremakinguseofAmazonWebServices,MicrosoftAzureandGoogleresources.Thescaleofthisusageisveryhardtodetermine;themaindriverisprobablymainlytheuseofvirtualisationtoallowworkflowsthatwerebuiltonlocalworkstationstoruneasilyonlargevirtualclusterswithouttheneedfortimeconsumingportingofworkflowstoclustersintheNeI.
NeiSurvey2015ProjectDirectorsGroup(PDG)
WhoaretheUK’snationale-Infrastructureproviders? 15
• HEIsaremakingincreaseuseofCloudresourcesforbackofficeandadministrativeservicesusedbytheirorganisations.
Recommendations:
ThenewlyestablishedRCUKCloudWorkingGroupshoulddevelopastrategytoestablishthescopeandpotentialofCloudservicesforresearchandinnovation.ThisshouldidentifywhichpartsoftheNeIcouldusefullyadopthybridCloudtechnologies,andhowCloudtechnologiescouldassisttheNeIinsharingresourcesbetweenitsdifferentprojects.
ToleverageUKinvestmentsinEUandinternationale-InfrastructuressuchastheEuropeanScienceCloud,EGIandEUDAT,RCUKshouldreportonthecurrentlevelofUKparticipationintheseprojectsanddeviseastrategytoensuretheUKbothmaximisesuse,andinfluencesthedesignandmanagement,oftheseinfrastructures.
NeISurveyReport-2015
ProjectDirectorsGroup(PDG)
SustainablefundingfortheNeIEcosystem
TheHEIProblem
TheadvantageofplacingservicesinHEIsisthatitplacesservicesinthesamelocationastheresearchcommunitiesthatusethem,soallowinggreaterinteractionbetweenserviceproviderandresearchers.
ItalsoprovidesanimportantlocalresourceforresearchersatHEIsandencouragestheentryofnewresearchersandresearchareasintousinglargescaledatafordiscoveryandsimulationservices,byprovidingbothlocalhelpandtraining.
HEIsarealsoallengagedinworkwithpublicsector,businessesandcommerce.TheyareanobviousinterfacewithSMEs.
TheHEIsaretheenginethatwilldrivetakeupofthesenewservicesandthereforeneedtobesuitablyresourced.Howeverthisisdifficulttoachieveinaco-ordinatedfashion.HEIseachhaveaparticularresearchmissionthattheywishtofulfilandtheir5yearplanswillbemarkedlydifferentfromeachother.
Nowthatthe2016-2021sciencecapitalfundingframeworkisinplaceandanE-InfrastructureRoadmapexistsanopportunitypresentsitselftodeviseasustainablefundingframeworkforHEIs.Thisshouldrecognisethat:
1. fECincomeintoHEIsshouldallowHEIstodelivercoresservicestotheirlocalresearchers.2. CapitalmoniesfromRCUKandothersourcescanbeusedtohelptoHEIsprovideservicesforboth
nationalprojectsandspecialistprojects.3. TheexistenceoftheNeISurvey,afundingframeworkandtheroadmapshouldallowNeIprovidersto
self-organiseand• createpartnerships/consortiatoprovideservices• poolandaggregateresourcesviaco-locationormakeuseofcloudservices• createservicesandtheneededcriticalmassofexpertisearoundparticularservicerequirements
orresearchareas• deliveragreaterrangeofservices,particularlyaroundindustrialandpublicsectorengagement,
researchsoftwareengineeringandtrainingservices.4. OperationalcostsremainanissueinthecurrentfundingclimateandHEIscanassistwithmeetingthis
issueifcapitalcanbeusedtofundtheirinfrastructure.
NeiSurvey2015ProjectDirectorsGroup(PDG)
NonHEIProviders 17
Recommendation:
HPC-SIGhasakeyroletoplayinfacilitatingNeIproviders’self-organisationintopartnerships/consortiatobuildcriticalmassofknowledge,achieveoperatingefficienciesandmaximiseproductivity,creatingeconomiesofscalewithoutcompromisingdiversity.
NonHEIProviders
TheResearchCouncilsandotherresearchorganisationshavemade/willbemakingtheircapitalplansandplanningfortheassociatedoperationalcosts.
Thereismeritinhavingcloserco-operationandknowledgesharingbetweenthoseengagedinprovidingservicesforourlargestprojects.Thiscanleadtoreductionsinprojectoperationalcostsandmakeprojectsmoreproductive.
Recommendations:
AneconomicsustainabilitymodelofthecurrentNeIshouldbeconstructedasamatterofurgencyandasustainabilitystrategydevelopedbythemainparticipantsoftheNeI.Workshopswillbeheldtodevelopthismodelandtodeveloppartnershipsinserviceprovision.ThisisnecessarytoassistwithplanningandtoallowcomparisonofservicecostswithPublicCloudproviders.E-Infrastructureprovidersneedtobegivenconfidencethatthereisalongtermplaniftheyaretoreleasefundsthatwillunderwritestaffrolesbeyondthelifetimeofcurrentprojects,andtodevelopacareerstructureforstaffworkingin“ResearchOperations”(ResOps).
RCUKshouldproposecommonresearchandinnovationrelatedmetricsforallNeIproviderstoensureastandardofreportingtoallowtheeffectiveness,efficiencyanddiversityofNeIserviceprovisiontobeassessed.
NeISurveyReport-2015
ProjectDirectorsGroup(PDG)
E-Infrastructureasasharedservice
ThepublicationofSirIanDiamond’sreport“Efficiency,effectivenessandvalueformoney”hasservedtoemphasisetheimpressivegainsthattheuniversitysectorhasmadeinimprovingequipmentsharingandengagementwithindustry.TheDiamondreportincludedexamplesofe-Infrastructureequipmentsharing(SES,UKDataService)andhighlightedthevalueofacross-researchcouncilinterestgrouptopromoteassetsharing.
FurtherbigdatainvestmentsbytheResearchCouncilshavecentredonsharedresourcesinregionalfocisuchasN8(LeedsAdvancedResearchComputing)andWales(Farr/ADRC-W).MRCinvestmentsinmedicalbioinformaticshaveresultedindevelopmentofthesharedeMedLabcluster(UCL,QMUL,LSHTM,Crick,Sanger,andEBI).ThisequipmentislocatedintheJiscSharedDatacentretogetherwithMRC-andNIHR-fundedequipmentbelongingtoImperialCollegeandKCL.RelocationofcomputeandstoragefrominadequatepremisesinLondontoamodernenergy-efficientdatacentreallowsHEIstofreeupspaceincentralcampusesandsavemoneyinpowercosts.
Thisconcentrationofco-locatedresourcesislargelyaresultbothoflargecapitalinvestmentswithlittlematchingoperationalcosts,butalsobecausethereisasmallpoolofpeoplewiththetechnicalskillsandknowledgerequiredtooperatetheselargecomputeanddatastoragesystems.Efficiencythroughco-locationisnotjustaboutsavingmoneybutalsoaconsequenceofthelackofsuitableskilledstafftobuildandsupportsystems.Co-locationandpartnershipisneededtoprovideeffectiveserviceprovisiontoacademicandindustrialusers.
ThissurveyhashighlightedthelargeuseofservicesbytheNHSandNHSrelatedprojects.ThisischieflyaconsequenceofnewinvestmentbytheMRCandDoHleadingtothesenewcommunitiesinteractingwiththeNeI.Thesewouldnothaverespondedtothe2014survey.
TheNeIisalreadyworkingtoprovidesecureaccessandsecuredataservicese.g.SafeShareforthesenewcommunities.
JischasestablishedwhatisbelievedtobethelargestVATcostsharinggroupintheUK,withover250institutionsparticipating:https://www.jisc.ac.uk/about/vat-cost-sharing-group.
Recommendations:
ServiceProvidersshouldbeencouragedtoformpartnershipsandpoolresourcestodelivereconomiesofscaleinprovidingafullrangeofservicesacrossacademic,industryandthirdsectororganisations.
Increasetrainingandupdateservicemanagementprotocolsappropriately,sothatthoseusingandmanagingNeIservicesareawareofthesecurityrequirementsoftheirprojectsandtheirsystemsrespectively.Thisshouldbedoneinaproportionatewaythatreflectsdatabeingprocessedanduses/usersofthesystems.
ExplorethepotentialofusingJisc’sVATcostsharinggrouptoreducethefrictionofsharingfacilitiessuchase-Infrastructure,andresearchequipmentmoregenerally.
NeiSurvey2015ProjectDirectorsGroup(PDG)
Industry 19
Industry
Engagementwithindustryisstilllargelyfocusedmostlyinafewsectors-advancedmaterialsandmanufacturing,energyandenvironment,andlifesciences.EngagementisdrivenbythetypeofindustrylocatedclosetotheNeIfacility.LocalEnterprisePartnerships,withtheirspecificexpertiseincertainindustrysectors,shouldprovideanidealroutetobuildrelationshipswithlocalbusinessesspecificallySMEs.
InparticularandelementofNeIinfrastructureshouldbepartoftheinfrastructureplanningbylargescaleregionalinitiativessuchastheNorthernPowerhouse,MidlandsEngine,WalesandScotland.Examplesofthiswouldbe:
• AnorthernJiscSharedDataCentre• ImprovedconnectivityforbusinessandacademiaoutsideoftheSE.
Recommendations
TheELCshouldorganisemeetingsbetweenregionalconsortia/partnershipsandNationalProjectswithLEPsandinitiativessuchastheNorthernPowerhouseandMidlandsEnginetoensurethatlocale-Infrastructureneedsareassessed.
IUKshouldworkwithregionalconsortia/partnershipsandNationalProjectstodeveloptheCatapultsandworkeffectivelywiththeresearchcommunityandgovernment.ExamplesincludethePrecisionMedicineCatapultwhichwillhaveitsHQinCambridgeandtheMedicalTechnologyCatapultwhichbelocatedclosetotheHartreeCentre.
KeyfindingsfromtheDowlingreport(BIS2015)aremirroredintheresponsestothesurvey.Effectivebrokeragebetweenindustry,particularlySMEs,andserviceprovidersiscritical.HEIsinparticularlackresourcetoidentifycommercialopportunitiesincontrasttothelarge/specialistandregionalfacilities(theHartreeCentrehas5dedicatedstaffforbusinessdevelopment).Wherethishasworkedeffectively,ithasbeenduetopartnershipwithinstitutionalbusinessdevelopmentteams.OneexampleisananalysisoftheopportunitiesandbarrierstoSMEuseofregionalcomputeresources,publishedbytheEPSRC-fundedCentreforInnovationinpartnershipwithUCLAdvances.AfurtherchallengeisthelackofawarenessonthepartofindustryoftheavailabilityandtypeofresourcesaspartoftheNeI.
TheDowlingreportalsonotedthatVATwasaparticularobstacletocollaborationandinnovationbetweenacademiaandindustryandrecommendedthat“thegovernmentneedstoaddresstheissueofVATonsharedfacilitiesasamatterofurgency”.
NeiSurvey2015ProjectDirectorsGroup(PDG)
Examplesofindustrycollaborations 20
Recommendations:
IndustryshouldhaveimprovedvisibilityofNeIresources,e.g.viaaccesstodashboardsandportalsdevelopedbytheNeIforitsownuse.
Regionalandnationalfacilitiesshouldhaveanindustrialengagementstrategyincludingmetricsthatpermittheeffectivenessofthestrategytobequantified.
LeverageexistingandnewRCUK-fundedbusinessdevelopmentstafftosupporte-Infrastructurerelatedelementsofindustryengagement,workingwithregionalcentresandlocalHEIs.EncourageuseofstrategicfundingsuchasImpactAccelerationAccountstopump-primebusinessdevelopmentactivitieswithinHEIsspecificallyforcomputeanddataresources.
Examplesofindustrycollaborations
TheEPSRCHPCMidlandssupercomputingcentreofexcellenceisprovidingaccessto£60MofsupercomputingequipmenttoRollsRoyce,facilitatedbyabrokerageschemedevelopedwithJisc.
TheSangerInstitutealsohascollaborations,andprovidesscientificITfor,on-sitespin-outcompaniessuchas14mGenomicsandCongenica.TheGenomeCampus(Hinxton)isbuildinga"BiodataInnovationCentre"nextyear,massivelyexpanding,withspacefor250peopleworkingonindustrial/commercialisationprojects.TheSangerInstitutewillbeprovidingtheITinfrastructurefortheCentre.
TheSquareKilometreArrayScienceDataProcessorprojecthousestheSKAOpenArchitectureLaboratoryinwhichIThardwareandsoftwarecompaniescanworkdirectlywiththeSDPteamtodeveloptechnologiesfortheSKAScienceDataProcessor.Highlightsincludelowenergycomputeclustersfordataprocessing(ARM),veryfastandcheapdiskarraysfordataintensivecomputing(DELL)andusingtheCloudoperatingsystemOpenstacktomanagetheSDPworkflows(Canonical).
OCF,aUKSME,hassupportedanOpenStackusergrouptoaccelerateuptakeofcloudtechnologiesbytheNeIcommunity,includingeMedLab,MRCCLIMBandtheCrickInstitute.
TheUKgovernmenthascommitted£113MtothefutureoftheHartreeCentre.Thisinturnhasleveragedupto£200MoftechnologyandonsiteexpertisefromIBM.FurthercasestudiesareavailablefromtheInnovateUKe-InfrastructureSpecialInterestgroup.
TheUKnowhosts7IntelParallelComputingCentres,whicharedesignedtodrivethedevelopmentofapplicationsandlibrariesonmanycoresystems.ThisisthelargestnumberofCentresoutsideoftheUSAand
NeiSurvey2015ProjectDirectorsGroup(PDG)
Examplesofindustrycollaborations 21
DiRAChasthelargestnumberofCentresofanyprojectintheworld.Activitiesvaryfromtheportingofapplications,buildingnewapplications,thedevelopmentoftrainingresources,dataimaginganddataanalysis,heterogeneousarchitectures,newmathslibrariesformanycoreprocessorsandthefinegainedmanagementofparalleljobs.TheseactivitieswillallowtheUKtobeamongthefirsttotakeadvantageofmanycoretechnologiesasitmatures.
Seehttp://www.intel.co.uk/content/www/uk/en/processors/xeon-phi/intel-parallel-computing-centers-overview-video.html.
DiRAChoststhreeoftheseCentresattheUniversitiesofEdinburgh,DurhamandCambridge(DAMTP).CentresarealsolocatedattheHartreeCentre,theUniversityofBristol,theEdinburghParallelComputingCentreandmostrecentlyImperialCollege.
TogiveaflavouroftheworkdoneasummaryoftheprojectscarriedoutbyDiRACisgivenbelow.TheseshowthatbyworkingonbeyondtheedgetechnologieswithfirmslikeIntelitispossibletoproducethesekindsofinnovativelowleveltasks/libraries.TheseprojectsalsosupportamixoftraditionalHPCandnewdatascienceapplications.
1.1
1. DevelopedtheGRIDdataparallellibrary,andcodestencils,toalloweasieruseofmatrixmethodsinparallelcodeonmanycoresystems(PeterBoyle,Edinburgh)
2. TestedanddevelopedmanycorevisualisationsoftwareusingXeonPhimanycoresystems-OSPRay(PaulShellard,Cambridge))
3. Developeddataanalyticsanddatamodellingworkflowsusingaheterogeneousprocessor(CPU)andmanycoreXeonPhisystemwhichwasbuiltintheUK(PaulShellard,Cambridge))
4. DevelopedthefinegrainedparalleltaskmanagementQUICKSCHEDlibrary,whichmakespoorlyscalingparallelcodescalemuchbetter.Itmanagestheso-calledloadbalancingproblemandisreallyneededifwewantcodestofunctionatpetascale(RichardBower,Durham).
5. ApplyingnovelstatisticalaccelerationtechniquestoBayesianMethodsandMarkovChainMonteCarlo–thusgreatlyreducingthetimesearchingparameterspace(MarkWilkinson,Leicester)andimprovingquantifiableuncertaintiesinimportantclassesofsimulations.
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixA-Listofsurveyorsandacknowledgements 22
AppendixA-Listofsurveyorsandacknowledgements
SusanMorrell(EPSRC)MartinHamilton(Jisc/NeIPDG)ClareJenner(UCL/DiRAC/NeIPDG)JackyPallas(UCL/eMedLab/Farr/NeIPDG)AlanReal(Leeds/N8/HPC-SIG)AndrewRichards(Oxford)JeremyYates(UCL/DiRAC/SKA/NeIPDG)
AppendixB-Whowillreceivethissurvey
TheNationale-InfrastructureProjectDirectorsGroupRCUKNationalE-InfrastructureGroupBISE-InfrastructureLeadershipCouncil.
AppendixC-Listofrespondents
ARCHIE-WeStCardiffUniversityCloudInfrastructureforMicrobialBioinformatics–MRCCLIMB-Birmingham,Cardiff,Swansea,CardiffCranfieldUniversityDiRAC@DurhamUniversityDiRAC@EPCCDiRAC@UniversityofCambridgeDiRAC@UniversityofCambridge(DAMTP)DiRAC@UniversityofCambridge(HPCS)DiRAC@UniversityofLeicesterDurhamUniversity
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixC-Listofrespondents 23
EMBL-EBIeMedLab(MRC)-UCL,QMUL,LSHTM,Crick,Sanger,EBIEPCCFarrNorth,HealtheResearchCentreHPCMidlandsHPCWalesImperialCollegeLondonKing'sCollegeLondonLancasterUniversityLoughboroughUniversityN8HPCNBIPartnershipLtdNorwichBioscienceInstitutes(TGAC,JIC,IFR,TSL)QueensUniversityofBelfastSES/CFISTFCHartreeCentreSTFCScientificComputingDivisionTheFrancisCrickInstituteTheInstituteofCancerResearchTheUniversityofBirminghamTheUniversityofNottinghamTheUniversityofSheffieldUVRI/MRCMedicalInformaticsCentreUniversityCollegeLondonUniversityofAberdeenUniversityofBathUniversityofBristolUniversityofCambridgeUniversityofEdinburghUniversityofExeterUniversityofGlasgowUniversityofLeedsUniversityofLeicesterUniversityofLiverpoolUniversityofManchesterUniversityofOxfordUniversityofPortsmouthUniversityofSouthamptonUniversityofStAndrewsUniversityofSussexUniversityofWarwickWellcomeTrustSangerInstitute
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixD–Servicemanagement:Thesurveyquestions 24
AppendixD–Servicemanagement:Thesurveyquestions
Q1OrganisationnameQ2OrganisationalunitQ3EmailaddressQ4Jobtitle
Budget
Q5IsyourHPCbudgetringfenced,ordoyouhavetomakeafreshcaseforsupporteachtime?Q6IsyourprimarybudgetforHPCCAPEXorOPEX?Q7AreyoudirectlychargedforpowerandcoolingorotherEstatescosts?Q8DoyoucapitaliseEstatescosts,orpaythemfromyourOPEXbudget?Q9Tellusaboutyourdatacentre(s)(selectallthatapply)Q10DatacentrePUE(ifknown)Q11Datacentrepowerdraw,ifknown
Projectmanagement
Q12Howistheprojectmanagementneededtoimplementanddelivernewprojectsprovidedatyourinstitution?Q13HowareProjectManagement(NewServiceImplementationandDelivery)resourcescalculated?Q14DoyouworkouttheresourcesneededintermsofFTE?Q15WhichofthefollowingbestapplytoyourHPC/BigData/ResearchComputing/ScientificComputingActivity?Q16DoyouusemetricstoworkouttheProjectManagementresourcesneeded?[itisassumedthesearechargedagainsttheCapitalCostoftheProject]Q17Arethey...[tickboxesforpercentagerangeofcapitalcostduetoProjectManagement]Q18WhichofthestatementsbestdescribehowyouarriveattheProjectManagementresourcesneededtodeliverandimplementtheproject?
Staffing
Q19HowmanyFTEsupportyourHPCactivity?Q20FTEcountif>4Q21HowmanywomenareinvolvedinHPCserviceprovisionatyourorganisation?Q22WhatmodelofHPCprovisionhasyourorganisationadopted?Q23BreakdownofeffortbyFTEQ24WomeninHPCservicemanagement(FTE)Q25Whatisyourapproachtotraining?PleasetickallthatapplyQ26URLforfurtherinformationontrainingifapplicableQ27Howwouldyouprefertopooltrainingacrossthesector?
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixD–Servicemanagement:Thesurveyquestions 25
Q28Whatproportionofyourresearchersareinself-supportingresearchgroups?Q29Whatarethemajortrainingchallengesandwhichdoyouaddresslocally?Q30OthertrainingchallengesQ31Otherformsofsupport.Doesyourorganisationdoanyofthefollowing?Q32AreyouabletoprovideHPCcasestudiesfor(e.g.)HPC-SIGwebsite,RCUKandInnovateUKe-InfrastructureSIG?
CloudandSharedServices
Q33Whichstatementsbestcharacteriseyourorganisation'suseof"cloud"services?Q34Useofcloudservices-furtherinformationQ35Whichstatementbestcharacterisesyourorganisation'sapproachtoHPCasasharedservice?Q36WhichareasofHPCprovisionmightyoushare?
Researchdatamanagement
Q37DoyouhaveaResearchDataManagementpolicy?Q38DoesyourorganisationhaveaResearchDataManagementservice?Q39URLforfurtherinformationaboutResearchDataManagementatyourorganisationifapplicableQ40WhattechnologieshaveyoudeployedthatcanberegardedasbeingforDataExploration(“BigData”)activities?
Academicimpact
Q41WhatareyourKeyPerformanceIndicatorstomeasureacademicimpact?Q42RefereedandconferencepaperspublishedQ43URLlistingthesepublicationsifapplicableQ44DoesyourorganisationproduceanAnnualReport?Q45DoesyourorganisationpublishResearchHighlights?
Whousesyourservice?
Q46DoyouprovideHPCservicesbeyondyourimmediateorganisation?Q47Whatservicesdoyouprovidetothirdparties?Q48HowmanyHEIsresearchgroupsuseyourservices?Q49PublicandthirdsectororganizationsQ50Whatpercentageofyourusersarefromindustry?Q51Ofindustryusers,whatproportionareSMEs?Q52WhatpercentageofsystemtimeisbeingusedbySMEs?Q53AreyoutakinganyspecificstepstoincreaseSMEuptake?Q54Whatsectorsarebeingservedbyyourindustrialusers?Q55Pleaseusethisspaceforanyfurtherinformationyouwouldliketoprovide
NeISurveyReport-2015
ProjectDirectorsGroup(PDG)
AppendixE–Servicemanagement:Summaryofthesurveydata
Wherefeasiblewehavetriedtobreakdownsurveyresponsesbylargeandspecialistfacilities,regionalcentres,andhighereducationinstitutions.Eventhen,manyHEIsclearlyoperateonashoestringe.g.withminimalstaffing,whereasothershavelargeteamsworkingonscientificcomputing-soitisnottrivialtodrawcomparisonsbetweenthethreegroups.WehaveincludedHPCWalesasa"regional"facilitybecauseofitsprimaryfocusonWelshresearchersandindustry.ItshouldbenotedthatHPCWaleswasnotpartoftheEPSRCregionalsupercomputercentreinitiative.
Budgetingmodel-Thevastmajorityofrespondents(73%ofthosewhoansweredthisquestion)indicatedthattheirprimarybudgetfore-Infrastructurewascapital(CAPEX).However,whilst63%statedthattheywerenotchargedforpowerandcoolingorotherEstatescosts,28%ofrespondentsindicatedthattheypaidEstatescostsfromOPEX-implyingthatsomeelementsoftheserviceprovisionbeyondsalarieswerebeingcoveredviaOPEX.
Thirdpartydatacentres-TheonlythirdpartydatacentrethatrespondentsreportedusingwastheJiscSharedDataCentre,inuseby8%ofinstitutionsresponding.Ofthoserunningin-housedatacentres,moststatedaPUEofbelow1.5.Severalrespondentsindicatedthattheywereonlyabletoreportdatacentrepowerdrawforabuildingoranentiredatacentrewhichtheirfacilitywaspartof.
Projectmanagementapproach-Itwasrareforrespondentstocallonthirdpartyprojectmanagementservicesfornewserviceimplementationanddelivery,withthisalmostalwaysbeingdonein-house,sometimes(13%ofcases)viadedicatedprojectmanagersbasedinITServicesdepartments.Respondentshadarangeofapproachestocalculatingtheirresourcerequirementsforamajore-Infrastructureproject,rangingfromhighlyformalisedto"backoftheenvelope".
68%ofrespondentsstatedthattheycalculatedstaffingcostsfortheirimplementationprojects.Itwasnotunusual(36%ofrespondents)forinstitutionstobidforadditionalstaffingaspartofane-Infrastructureprojectproposal.Onlyoneinstitutionusedmetricstoworkoutprojectmanagementresourcesneeded,and46%ofrespondentsagreedwiththestatementthat"wedon'tcalculatetheseresources,itjusthappens".
Staffing-24%ofrespondents(includingHEIsandlargeandspecialistfacilities)statedthattheyhadtwoorlessfulltimeequivalentemployeessupportingtheire-Infrastructureactivities.Conversely,9out25HEIsresponding(36%)hadteamsoffourormoreemployeesworkinginscientificcomputing.OneHEIreported23FTEsworkingonitsservices,buttheselargerstaffingcomplementsweretypicallyassociatedwithregionalandlarge/specialistfacilities.
WomeninHPC-thesurveyshowedthattherewere55womeninvolvedinserviceprovisionacrossallofthefacilities,mainlyworkingparttime.ThesituationwasparticularlydisappointinginHEIs,withonly14womeninvolvedinserviceprovisionininstitutions.Manyrespondentsstatedthattheywereunsurehowmanywomenwereinvolvedintheirservices,whichisoddgiventhesmallteamsizesatalargeproportionofHEIs.AseparatesurveyhasbeenconductedofARCHERandDiRACserviceusers,togaugethesituationfromtheuser'sperspective.
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixE–Servicemanagement:Summaryofthesurveydata 27
Training-36%ofrespondentsreportedthattheycarriedoutintroductorytrainingcourses.Thiscoversjust40%ofHEIrespondents.Only24%ofHEIsrespondingoperatedanadvancedtrainingcourse,andonly34%ofHEIsactivelypromotedtrainingavailablefromothersourcese.g.ARCHERandapplicationvendors.OnlytwoinstitutionsprovidedtrainingaspartofaDoctoralTrainingProgramme,and3aspartofaCentreforDoctoralTraining.However,therewaswidespreadinterestinsharingtrainingmaterialsfreelyonline(73%ofrespondents)and/orcreatingamarketplacefortrainingmaterialsanddeliveryproviders(34%ofrespondents).
Skills-10outof25HEIsresponding(40%)reportedthatlessthanhalfoftheiruserswereinself-supportingresearchgroups.HEIs,regionalcentresandlarge/specialistprovidersallsawarangeoftrainingchallengesratherthanasinglekeyissue-achangefromthe2014surveywhereLinuxorientationemergedasahugeproblem.WewillfollowthisdevelopmentupwithHPC-SIGmemberstoseewhethertherequirementforLinuxskillshasdiminished,oritisbecomingmorecommonfornewserviceuserstoalreadybefamiliarwithLinux.
Communitybuilding-Institutionsreportedthattheywereaugmentingtrainingwitharangeofadditionalcommunityandexpertisebuildingapproaches,includinginformalnetworkingforserviceusers(68%ofHEIs),wikisoronlineforums(48%ofHEIs),andpublicspeakingatresearcherfocussedevents(40%ofHEIs)ordedicatedscientificcomputingevents(36%ofHEIs).Manyserviceproviders(24%ofHEIsand50%oflargeandspecialistfacilities)werealsoabletoprovidecasestudiesofhowtheirservicehadbeenused.
ResearchDataManagement-76%ofrespondentssaidthattheyhadaResearchDataManagementpolicy,with4HEIsreplyingthattheywereeitherunsureaboutthisordidnothaveapolicyatpresent.51%ofrespondentshadapilotorproductionRDMserviceatthetimeofrespondingtothesurvey,with5unsure.
Academicimpact-commonKeyPerformanceIndicatorswerethenumberofusersoftheservice(63%ofrespondents),thenumberofresearchpaperspublished(61%),thenumberofprojectsrunonsystems(51%)andtheamountofresearchgrantfundingbroughtin(51%).However,onlyahandfulofrespondents(e.g.4HEIs)wereabletolistthenumbersofpublishedpapersthathadusedtheirsystem,andonlyasmallnumberofrespondentswereproducinglistsofpublishedpapers,anannualreportorresearchhighlights.
Crosssectorandindustrialimpact-manyrespondentswereinvolvedininter-institutionalcollaborationssuchastheEPSRCregionalcentres,theFarrInstitute,GridPPandtheMedicalBioinformaticsInitiative.51%ofrespondents,including32%ofHEIs,providedcomputeandstorageservicestothirdparties.27%ofrespondentsworkedwiththeNationalHealthService.Industrialuseofnationale-Infrastructurefacilitieswaslargelyconfinedtoasmallnumberofproviders,withsomeHEIsoutstrippinglargeandspecialistandregionalcentres.Just7providersreportedthattheywereworkingwithSmalltoMediumsizeEnterprises(SMEs).
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixF–Servicemanagement:Fullbreakdownofsurveydata 28
AppendixF–Servicemanagement:FullbreakdownofsurveydataBudget
Q5IsyourHPCbudgetringfenced,ordoyouhavetomakeafreshcaseforsupporteachtime?
HEILarge/
Specialist Regional GrandTotal
FreshcasetoOrganisation 12
12
Partlyringfenced 8 2 1 11
FreshcasetoExternalFunders
5 1 6
Whollyringfenced 4 2
6
Other 1 3 1 5
GrandTotal 25 12 3 40
Q6IsyourprimarybudgetforHPCCAPEXorOPEX?
HEI Large/Specialist Regional GrandTotal
CAPEX 18 9 2 29
Other 3 2 1 6
OPEX 4 1 5
GrandTotal 25 12 3 40
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixF–Servicemanagement:Fullbreakdownofsurveydata 29
Q7AreyoudirectlychargedforpowerandcoolingorotherEstatescosts?
HEI Large/Specialist Regional GrandTotal
No 19 4 2 25
Yes 5 5 1 11
Other
3
3
Don'tknow 1
1
GrandTotal 25 12 3 40
Q8DoyoucapitaliseEstatescosts,orpaythemfromyourOPEXbudget?
HEI Large/Specialist Regional GrandTotal
19 7 2 28
OPEX 6 4 1 11
Other
1
1
GrandTotal 25 12 3 40
Q9Tellusaboutyourdatacentre(s)(selectallthatapply)
HEILarge/
Specialist Regional GrandTotal
Weoperatein-housedatacentres
24 12 3 39
WeusetheJiscDataCentre 1 2
3
Weuseathirdpartydatacentre
GrandTotal 25 14 3 42
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixF–Servicemanagement:Fullbreakdownofsurveydata 30
Q10DatacentrePUE(ifknown) Q11Datacentrepowerdraw,ifknown
Projectmanagement
Q12Howistheprojectmanagementneededtoimplementanddelivernewprojectsprovidedatyourinstitution?
HEI Large/Specialist Regional GrandTotal
Wedosomeaspectsoftheprojectmanagementusingourin-housesystemsteamsandtherestisprovidedbythesuccessfulvendor
12 4 3 19
WedoALLaspectsoftheprojectmanagementusingonlyourin-housesystemsteams
6 7
13
OurcentralITdept.hasprojectmanagementstaffandwecanusethemifweneedto
4 1
5
WepayexternalcontractorstohandleALLaspectsoftheprojectmanagement
1
1
GrandTotal 23 12 3 38
Q13HowareProjectManagement(NewServiceImplementationandDelivery)resourcescalculated?
• SupportedbyanSTFCgrantandarebundledwiththeoverallDiRACproject• ProjectManagementresourcesarecalculatedonaneedsbasis,andcurrentlywerequireoneFTE,whichisfunded
fromtwodifferentsources• Wemakeanestimateofallstaffeffort(FTEs)foreachproject,attheprojectproposalstage.• Weuseour25yearsexperiencewhenwritingproposalsandtenderstoprovideappropriateresourcesforeachproject
01000200030004000500060007000
Q11.Datapowerdraw(kW)
Large/specialist
Regional
HEI
0
0.5
1
1.5
2
2.5
Q10.DatacentrePUE
Large/specialist
Regional
HEI
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixF–Servicemanagement:Fullbreakdownofsurveydata 31
andservice.• Weareattheinitialstart-upphase,so100%ofthetimehasthusfarbeenonnewserviceimplementationand
delivery.• BytheBusiness&Deliveryteams,baseduponanextensivespreadsheet• Basedonourexperienceandexpertiseandthecustomerrequirements• BasedonstandardFTEcosts• Veryinformally;foreachprocurement,oneoftheHPCarchitectswilltakeonprojectmanagementresponsibilityfor
thenewservice.• EachProjectandassociatedservicesaresubjecttoaninitialreviewbytheOperationalandUserServicesGroup• ProjectmanagementcostsfortheimplementationofanewHPCserviceareincludedinthesystemprocurement• Nodirectchargingsincetheseareinternalresources• Itisbasedonthesizeofthecontractandestimateddeploymenttime• Dayratebasedonactualcostsblendedacrossamixofcorepermanentstaffandcontractors• FormalProjectManagementisonlyrequiredonanadhocbasissotheresourcecalculationsaresimilarlydoneonan
adhocandcasebycase• Aspartofourprojectdeliveryprocesses.• Advisefromin-houseprojectdeliveryteams,typically1PMpermajorproject.• PartoftheSysAdminsjob• Resourcescalculatedinternally
Q14DoyouworkouttheresourcesneededintermsofFTE?[ItisassumedthesearechargedagainsttheCapitalCostoftheProject]
HEI Large/Specialist
Regional GrandTotal
No 6 1
7
Yes 6 8 1 15
GrandTotal 12 9 1 22
Q15WhichofthefollowingbestapplytoyourHPC/BigData/ResearchComputing/ScientificComputingActivity?
HEILarge/
Specialist Regional GrandTotal
TheyarecalculatedintermsofrequiredFTEandsomeofthisistensionedagainstouroverallstaffresourceallocationandtherestisrequestedintheProjectTender.
2 5 1 8
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixF–Servicemanagement:Fullbreakdownofsurveydata 32
TheyarecalculatedintermsofrequiredFTEandthisistensionedagainstouroverallstaffresourceallocation.
3 3
6
Unsure 2 2
4
TheyarecalculatedintermsofrequiredFTEandsomeofthisistensionedagainstouroverallstaffresourceallocationandsomeisrequestedfromourcentralITdept.
1
1 2
TheyarecalculatedintermsofrequiredFTEandallofthisHAStobeprovidedbyourcentralITdept.
1
1
TheyarecalculatedintermsofrequiredFTEandallofthisisrequestedfromourcentralITdept.
1
1
GrandTotal 10 10 2 22
Q16DoyouusemetricstoworkouttheProjectManagementresourcesneeded?[itisassumedthesearechargedagainsttheCapitalCostoftheProject]
HEI Large/Specialist Regional GrandTotal
No 9 2 1 12
Yes 1 1
GrandTotal 10 2 1 13
Q17Arethey...
Lessthan5%ofthetotalcapitalcostsBetween5and9.99%ofthetotalcapitalcostsBetween10and14.99%ofthetotalcapitalcostsBetween15and19.99%ofthetotalcapitalcostsOver19.99%ofthetotalcapitalcostsUnsure
Oneinstitutionrespondedtosaythattheirprojectmanagementcostswerelessthan5%oftotalcapitalcosts.
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixF–Servicemanagement:Fullbreakdownofsurveydata 33
Q18WhichofthestatementsbestdescribehowyouarriveattheProjectManagementresourcesneededtodeliverandimplementtheproject?
HEILarge/
Specialist Regional GrandTotal
Wedon'tcalculatetheseresources-itjusthappens
4 1 1 6
Wedon'tcalculatetheseresourcesbecausetheseresourcesarealreadyincludedinouroverallservicedeliveryplans.
4 1
5
Alreadyincluded 1 1
Unsure 1 1
GrandTotal 10 2 1 13
Staffing
Q19HowmanyFTEsupportyourHPCactivity?Pleasechoosethenearestanswerandentertheexactfigurebelowif>4
HEILarge/
Specialist Regional GrandTotal
1 2 1 3
1.5 3 1 4
2 2 2
2.5 5 1 1 7
3 2 3 5
3.5 1 1
4 3 1 4
>4 6 5 11
GrandTotal 24 11 2 37
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixF–Servicemanagement:Fullbreakdownofsurveydata 34
Q20FTEcountif>4 Q21HowmanywomenareinvolvedinHPCserviceprovisionatyourorganisation?
Q22WhatmodelofHPCprovisionhasyourorganisationadopted?
HEI Large/Specialist
Regional GrandTotal
CentralITdepartment 2 3 5
DevolvedtoFacultiesandSchools
11
Governmentresearchestablishment
2 2
Independentcentre 2 2
Mixtureofdevolvedandcentral
4 4
Other 1 1
Regionalornationalfacility 1 1 2
GrandTotal 7 8 2 17
0
10
20
30
40
50
60
70
80
Q20.FTEcountif>4
Large/specialist
Regional
HEI
0
5
10
15
20
Q21.WomeninHPCprovision
Large/specialist
Regional
HEI
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixF–Servicemanagement:Fullbreakdownofsurveydata 35
Q23BreakdownofeffortbyFTE
89
46
42
13
1
02468
10
Q23.Effortforsystemsupport
13
41 1
41
0
5
10
15
Upto0.5FTE
0.5-1FTE
1to1.5FTE
3to4FTE
Over5FTE
Unsure
Q23.Effortforsonwareengineering
20
41 2 2 2 1
0510152025
Q23.Effortforusertraining
18
4 31 2 2 1
0
5
10
15
20
Q23.Effortforprojectmanagement
22
4 3 2 1 2 10510152025
Q23.Effortforstrategicengagement
16
7 52 2 1
41
05
101520
Q23.Effortforapplicaoonsupport
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixF–Servicemanagement:Fullbreakdownofsurveydata 36
Q24WomeninHPCservicemanagement(FTE)
5
32
1
3
0123456
Upto0.5FTE
0.5to1FTE
1to1.5FTE
1.5to2FTE
Unsure
Q24.Womeninsystemsupport
3
2
1 1
3
00.51
1.52
2.53
3.5
Upto0.5FTE
1to1.5FTE
3to4FTEOver5FTE Unsure
Q24.Womeninsonwareengineering
6
21 1
3
0
2
4
6
8
Upto0.5FTE
1to1.5FTE
2to3FTE
Over5FTE
Unsure
Q24.Womeninapplicaoonsupport
6
2 21
2
0
2
4
6
8
Upto0.5FTE
0.5to1FTE
1to1.5FTE
3to4FTE
Unsure
Q24.Womeninusertraining
16
1 1 1 1 3 2
05101520
Q23.Effortforotherusersupport
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixF–Servicemanagement:Fullbreakdownofsurveydata 37
TrainingandUserSupport
Q25Whatisyourapproachtotraining?Pleasetickallthatapply
HEI Large/Specialist Regional GrandTotal
Werepurposeexistingtrainingmaterialsdevelopedelsewhere
12 5 2 19
Wedevelopourowntraining 9 4 2 15
Werunourownintroductorytrainingcourses
10 3 2 15
Wepromotetrainingavailablethroughe.g.ArcherCSEandapplicationvendors
8 2 1 11
Werunourownadvancedtrainingcourses
6 4 0 10
8
1 12
0
2
4
6
8
10
Upto0.5FTE
1.5to2FTE
4to5FTE Unsure
Q24.Womeninotherusersupport
9
31 1
3
0246810
Upto0.5FTE
0.5to1FTE
1to1.5FTE
3to4FTE
Unsure
Q24.Womeninstrategicengagement
5
21 1
3
0
2
4
6
Upto0.5FTE
0.5to1FTE
1to1.5FTE
4to5FTE
Unsure
Q24.Womeninprojectmanagement
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixF–Servicemanagement:Fullbreakdownofsurveydata 38
WeprovidetrainingaspartofaCentreforDoctoralTraining 3 1 1 5
WeprovidetrainingaspartofaDoctoralTrainingProgramme 2 1 1 4
Other 0 1 1 2
Wedonotprovidetraining 0 0 0 0
Q26URLforfurtherinformationontrainingifapplicable
• https://virgodb.cosma.dur.ac.uk• http://www.dirac.ac.uk/training.html• https://www.epcc.ed.ac.uk/education-training,http://archer-www.epcc.ed.ac.uk/training/• http://www.hpcwales.co.uk/solutions/skills-and-training• http://www.shef.ac.uk/cics/research/training• http://www.cardiff.ac.uk/arcca/services/events/index.html• http://www.kcl.ac.uk/hpc/services/training.aspx• http://www.ucl.ac.uk/isd/services/research-it/training• https://www.acrc.bris.ac.uk/acrc/training.htm• https://www.wiki.ed.ac.uk/display/ecdfwiki/Courses+and+Events• http://wiki.rac.manchester.ac.uk/community/Courses• http://www.arc.ox.ac.uk/content/training• Werunourownintroductorytrainingcourses• http://www.sussex.ac.uk/its
Q27Howwouldyouprefertopooltrainingacrossthesector?
HEILarge/
Specialist Regional GrandTotal
Sharematerialsonlinefreelyanddeliverlocally
21 8 1 30
Marketplaceoftrainingmaterialsandcoursedeliveryproviders 8 4 2 14
Other... 1 1 1 3
Donotfeelthiswouldbebeneficial
1 2 0 3
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixF–Servicemanagement:Fullbreakdownofsurveydata 39
Q28Whatproportionofyourresearchersareinself-supportingresearchgroups?
HEI Large/Specialist Regional GrandTotal
Noneornotknown
6 1
7
<=25% 4
4
25%to50% 6 2 2 10
50%to75% 3 1 1 5
>75% 6 7
13
GrandTotal 25 11 3 39
Q29Whatarethemajortrainingchallengesandwhichdoyouaddresslocally?
HEILarge/
Specialist Regional GrandTotal
Generalintroductiontoprogramming
2 4 2 4
Targettedprogrammingadvice,e.g.usingMPI,PE
3 4 2 4
Datascience 4 4 2 4
Other.. 4 3 1 4
Linuxorientation 1 3 1 3
Applicationspecificadvice 3 3 2 3
Q30Othertrainingchallenges
• Codeoptimizationandportingtonewtechnologies.• AswellasaddressinglocaltrainingchallengesweaimtospreadtrainingexpertiseacrosstheUKandbeyond.All
materialsareCClicensed,weengagewithothercentrestohelptraintheirtrainers,wedeveloponlinetraining
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixF–Servicemanagement:Fullbreakdownofsurveydata 40
materialsthatcanaccessedfromanywhereatanytime.• Trainingthetrainersi.e.enablingscientificcommunitiestosupportthemselves.• Bioinformaticsandgenomicsskillsforscientists.Resourcingpressurehasmadethisdifficulttoprovidelocally.• ContinualchallengeastothebestROI-tensionbetweentargetingthegrowthofnewusercommunities(resource
intensive)andfocusingontheestablishedresearchersandenhancingtheiroutput(publicationsetc.)-thelatterrequiresdomainexpertisebythetrainers.
• WhethertoimplementchargingtensionedagainstuptakebyinitialresearcherinterestHowtobestmonitortheimpactofthetrainingthroughsubsequentfollow-up.Multi-stagequestionnaire/survey-immediatelyposttrainingand3monthsafter.
• TrainingbiologiststouseGalaxy.• IntroductiontotheHPCcluster-addressingscheduler,storageandbestpractices• Helpinguserscopewiththechangesinworkflowinmovingfromdesktopcomputingtoamorebatchoriented;central
HPCsystem,includingtheprogressionfromGUIstoamorecommandlineenvironment.Thiscantakeagooddealofeffortwithsomenewusers.
• DeliveringwhatlookslikebespoketrainingtoCDTsisachallenge.Asiscoordinatingregionalandnationaleffortstoshareresources.Inrealityweprovideabroadrangeoftrainingfrommanydifferentsources.Wealsoembracetrainingdeliveredbyvendorstoourresearchers.
• Informationaboutavailablesoftware(incl.visualisationandanalysis).Examplebatchjobs.Occasionallyface-to-facetraining.
Q31Otherformsofsupport
HEILarge/
Specialist Regional GrandTotal
InformalnetworkingforexistingHPCusers
17 7 3 27
HPCcommunitywiki/forum 12 10 2 24
SeminarseriesfeaturingHPCbasedresearch
9 4 2 15
Speak/exhibitatresearcherfocussedevents
10 3 2 15
EmbedHPCstaffinresearchgroups
7 7 0 14
DirectworkonresearchprojectsbyHPCstaff
8 6 0 14
Other. 2 3 0 5
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixF–Servicemanagement:Fullbreakdownofsurveydata 41
Q32AreyouabletoprovideHPCcasestudiesfor(e.g.)HPC-SIGwebsite,RCUKandInnovateUKe-InfrastructureSIG?
HEILarge/
Specialist Regional GrandTotal
No 16 5 21
Yes 6 6 3 15
GrandTotal 22 11 3 36
CloudandSharedServices
Q33Whichstatementsbestcharacteriseyourorganisation'suseof"cloud"services?
HEILarge/
Specialist Regional GrandTotal
Someofourresearchersusecloudservicesinanunsupportedway 14 4 1 19
Wedon'tusecloudservicesatall 6 2 2 10
Someofourresearchersusecloudserviceswithcentralsupport
4 3 0 7
Other.... 2 3 1 6
Wehaveapolicyonuseofcloudfacilities
0 1 0 1
Thereareothercloudproviderswewouldliketoseethesectorpartnerwith 1 0 0 1
WeuseJisc'sclouddeals,e.g.Amazonportal
1 0 0 1
Mostofourresearchersusecloudserviceswithcentralsupport
0 0 0 0
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixF–Servicemanagement:Fullbreakdownofsurveydata 42
Weexclusivelyusecloudbasedcomputefacilities 0 0 0 0
Q34Useofcloudservices-furtherinformation.Pleasedescribethecloudservicesyourorganisationuses,orwishestouse.
• Weusegithubforsoftwareversioncontrolandpromoteits(orequivalentalternativeservice)toourusers.Manyusersusecloudservicesontheirown,too,e.g.dropboxetctosharefilesbutthisistotallyunsupportedandnotrecommended.
• Diracfacilitiesareprovided"asaservice"fromthepointofviewoftheendusers.AlthoughthisisnotususingHPCasaService,itisusprovidingHPCasaService.
• EMBL-EBIEmbassyCloud-VMware&OpenStack• http://www.fortissimo-project.eu/Webservicesforbigdata:EUDATInvestigatinguseofStarClusterforspecificHPC
workloadsinthecloud• OccasionalAmazonS3atsmallscale.VMsforad-hocwebservers(shortlived).HostedVMsfordeliveringoff-site
training.• WeprovideprivatecloudtoouruserbaseusingavarietyoftechnologiessuchasVCloud,OpenNebula• LargescaleuseofOpenStackforresearch.VMWareusedtosupportbusinessactivities.• AWSatthemoment.We'dliketobeusingOpenStackorsimilarourselves,toprovideanAWS-likeexperiencetothe
users.• WehavemadeanumberofeffortstolaunchourownCloudservicestogetherwithFujitsu.Thisrequiresanenhanced
HPCenvironment(security,easeofusewithdomainspecificportals,billingandSLAcontractualexpertise)thatarenottypicallynotrequiredbyanacademic
• WehaveevaluatedAWS(includingextendingourclusterusingBrightclustermanager),butfounditunsuitableformanyHPCapplications.
• Somead-hocusageofAmazoncloudviaComputerScienceresearchergroups.UndertakingacostandbenchmarkingperformanceanalysisincollaborationwithBioScienceinQ3/2015Annualcostperformanceanalysisofcloudvs.in-houseprovisionpresentedtoGovernanceGroup
• Wewouldbeinterestedinsomeformofcloud-burstingforcompute-orpossiblySaaSforsomeproductswhichwouldrelievethemainHPCsystemofthesesmallerusers.
• WearealsoaspecialistHPCcloudserviceinourownright.• Someresearches(e.g.computerscientists)useIaaScloudservicesinanunsupportedway.Wehavealsointerfaced
ourHPCtoAmazonAWSfor"cloudbursting"experimentallybuthavenotfoundtheperformance/priceratiotobebeneficialatthistimeforrawHPC.
• Office365,Azure,Simplicity• AmazonGlacier.Arkivum.• Weusecloudforthingslikeemail&someworkingdatastore.We'reinconversationwithMicrosoftabout
opportunitiesinscalar/throughputprocessing.• Weareinterestedinexploringwhatadditionalfeaturesofourexistingservices,oradditionalservices,cloudresources
wouldenable.• WeareinvestigatingtheeasewithwhichuserscanpotentiallyexploittheAmazonportalviatheJiscdeal.• Amazon,MSAzureetc.Theuseoftheseservicesismostlyunsupportedbutwehaveanapproachthatgivessupport
toresearchnomatterwhatplatformtheyuse.Cloudispartofthat.
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixF–Servicemanagement:Fullbreakdownofsurveydata 43
• UseofAmazon,MSAzure-especiallywherefreeaccesshasbeengrantedtosupportprojects.In-house'cloud'typeservicesusedfordeliveryofsomeHPCrelatedcomputations-mainlyfromVMWarebasedsystems
• GoogleDrive• Developingprivatecloudandcloudbursting.• Local/privatecloud;wedonotusepublicclouds(atleastnotinacentrally-supportedway)
Q35Whichstatementbestcharacterisesyourorganisation'sapproachtoHPCasasharedservice?
HEI Large/Specialist Regional GrandTotal
WeprovideaccesstoHPCasasharedservice,e.g.EPSRCregionalHPCcentres
7 9 3 19
Wemaybeinterestedinexploringsharedservicesinthefuture 6 1
7
Notapplicable 3 1 4
Weusesharedservicesprovidedforusbyothers 4
4
Other 3 3
Q36WhichareasofHPCprovisionmightyoushare?
HEILarge/
Specialist Regional GrandTotal
Training 12 4 3 19
Applicationsupport 10 5 3 18
Systemmanagement 10 6 0 16
End-to-endserviceprovision 7 5 2 14
Datacentre 5 7 1 13
Softwaredevelopmentassistance
8 4 1 13
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixF–Servicemanagement:Fullbreakdownofsurveydata 44
Otherusersupport 7 4 1 12
Other 1 1 0 2
Researchdatamanagement
Q37DoyouhaveaResearchDataManagementpolicy?
HEI Large/Specialist Regional GrandTotal
No 2 1 3
Unsure 2 3 5
Yes 21 7 3 31
GrandTotal 25 11 3 39
Q38DoesyourorganisationhaveaResearchDataManagementservice?
HEI Large/Specialist
Regional GrandTotal
Notyet 6 6 12
Pilotservice 9 1 10
Productionservice 6 3 2 11
Unsure 3 2 5
GrandTotal 24 11 3 38
Q39URLforfurtherinformationaboutResearchDataManagementatyourorganisationifapplicable
• http://www.lib.cam.ac.uk/dataman/• https://www.epcc.ed.ac.uk/facilities/uk-research-data-facility• https://www.stfc.ac.uk/1930.aspx• AtthemomentweadoptRDMpracticesfromourstakeholderinstitutions.• http://www.strath.ac.uk/researchdataproject/• http://www.sheffield.ac.uk/library/rdm
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixF–Servicemanagement:Fullbreakdownofsurveydata 45
• http://www.nottingham.ac.uk/research/research-data-management/index.aspx• http://www.cardiff.ac.uk/insrv/researchdata/managingdata/index.html(Converis-basedsolution)• http://www.kcl.ac.uk/library/researchsupport/research-data-management/index.aspx• http://www.lancaster.ac.uk/library/rdm/• http://www.lboro.ac.uk/service/research/offcampus/rdm.htm• https://www.qub.ac.uk/directorates/ResearchEnterprise/ResearchPolicy/ResearchDataManagementPolicy/• http://www.bath.ac.uk/research/data/• http://data.bris.ac.uk/• http://www.ed.ac.uk/schools-departments/information-services/research-support/data-management• http://www.liv.ac.uk/csd/research-data-management/• http://tinyurl.com/owpwj2o• http://researchdata.ox.ac.uk/• http://www.st-andrews.ac.uk/itsupport/academic/research/about/strategy/• http://www.sussex.ac.uk/library/research/researchdatamanagement/
Q40WhattechnologieshaveyoudeployedthatcanberegardedasbeingforDataExploration(“BigData”)activities?
• Largesharedmemorycomputers;100-terabyte-scalefile-systems• HTE,Spark/Hadoop• DIRMachine(DataIntensiveVMCluster)RDFCluster(LinuxContainersbasedclusterforDataAnalytics)SPRINT(MPI
ParallelR,http://www.r-sprint.org/)SGIUV20• 00(DigitalHealthInstitute,forcomplexdataanalytics• IBMproducts:BigInsights,Streams,InfosphereDataExplorer,SPSS,InfosphereContentAnalytics,Cognos,Watson• SCDmanageslargescalescientificdatainorderof50PBwhichisaccessibleathighdataratesusingarangeof
technologiesappropriatetothecustomer.e.g.for• JASMINusingPanasaswithamultiterabitIOcapacity.GridPPCastorusingcommoditydisksolutionstoSL85000tap• Tieredstorage,lookingatobjectstorageforcapacityinthenextiteration.• Tableau.RforGenome-WideAssociationStudies(ifthosecountas"BigData")• Hadoop,Cassandra,hive,pig,spark,maven,NoSQLdatabases.• Afilestorededicatedtoresearchdata,abaselevelserviceisfreeofcharge.Researchgroupscanpurchaseadditional
storage.ThefacilitylinkstotheHPCfacility.• Preparinginfrastructure(storage,networks)tosupportfuture'BigData'research.• Galaxywebservice,Hadoop,NoSQLdatabases.• Nothingatpresent.WillbelookingintoHadoopinthenearfuture• Wehavefundedasmallhadoopclusterasatestbed.We'relookingatimagelibrarytechnologies(mainlyformedical
researchprojects)• Hadoop• GPFS,Aridhia,largememorysystems• pbdRsupportedoncluster;sitelicense(viaEngineering)forTecPlotvisualisationsoftwareaswellasseveralopen
sourcedataexplorationandvisualisationtools.• Wehavespecificallydeployedalargesharedmemoryfacilityinsupportofanalysisofpublichealthrecords(Farr
InstitutefacilityaspartofN8HPC)andwehaveadataanalysisclusterdelayedinstitutionally.• nonedirectlyyetbyARC.Otherswithintheuniversityexploredifferenttechnologiessuchashadoop
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixF–Servicemanagement:Fullbreakdownofsurveydata 46
• Small7nodemaprinstallation.• InstalledRforchem/bioinformatics• SQLservices• Dataanalyticscluster(Hadoop;streaminganalytics);localcloudforbioinformatics
Academicimpact
Q41WhatareyourKeyPerformanceIndicatorstomeasureacademicimpact?
HEILarge/
Specialist Regional GrandTotal
Amountofresearchintermsofresearchgrants 14 4 3 21
Numberofdifferentresearchareasusingthesystems
15 3 2 20
Numberofpost-gradandpost-doctoralusers 6 3 2 11
Numberofprojectsrunonsystems 13 5 3 21
Numberofresearchpaperspublished 14 8 3 25
Numberofthesesproduced 2 2 3 7
Numberofusers 18 5 3 26
Other...... 4 2 1 7
Researchhighlights-highprofilepapers,breakthroughs,newsitems
10 7 2 19
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixF–Servicemanagement:Fullbreakdownofsurveydata 47
Q42Refereedandconferencepaperspublishedinthelastyear
Q43URLlistingthesepublicationsifapplicable
• http://www.cosmos.damtp.cam.ac.uk/info/cosmos-publications-2014/• http://www.dirac.ac.uk/science_all.html• Separatereportavailableonrequest.• https://pure.strath.ac.uk/portal/en/equipment/uoshpc(a35d23ad-d4fa-481c-b7db-e535add521f7).html• Containedinaseparatereport-availableuponrequest• Seebelowreport
Q44DoesyourorganisationproduceanAnnualReport?Ifso,pleaseenteritsURLbelow
• COSMOSannualreportsarepartoftheDiRACAnnualReportsathttp://www.dirac.ac.uk/• http://www.dirac.ac.uk/science_all.html• http://www.ebi.ac.uk/about/brochures• Differentreportsfordifferentprojectsandservices• http://www.stfc.ac.uk/SCD/resources/PDF/SCD_Science_Highlights_2014.pdf• Inprogress• http://cics.dept.shef.ac.uk/reports/cics-annual-report-2013.pdf• http://www.cf.ac.uk/arcca/news/annualreport2015.html(inpress)• http://www.kcl.ac.uk/newsevents/publications/report.aspx• OurannualreportisaninternaldocumentandpresentedtoourHPCBoard.However,wedoproducepublicreports
usuallytocoincidewithanewsystem,e.g.:https://www.acrc.bris.ac.uk/acrc/HPC_report.pdf• http://www.cam.ac.uk/annual-report• https://www.liv.ac.uk/annual-report/• http://n8hpc.org.uk/n8-hpc-annual-survey-2014-released/
0
0.5
1
1.5
2
2.5
0 50 100 150 200 250 300
Numberofrespondents
Q42.Refereedandconference paperspublished
HEI
Large/Specialist
Regional
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixF–Servicemanagement:Fullbreakdownofsurveydata 48
• http://www.it.ox.ac.uk/about/reports/it-services-annual-report-20132014
Q45DoesyourorganisationpublishResearchHighlights?IfsocanyouprovideURLsforupto3recenthighlights
• http://www.cosmos.damtp.cam.ac.uk/info/research-highlights-2013-14/• http://www.dirac.ac.uk/science_news.html• Differentfordifferentprojectsandservices• ForSCARF:http://www.scarf.rl.ac.uk/sites/default/files/docs/RAL-TR-2014-017.pdf• https://www.sanger.ac.uk• Yes-toappearonnewwebsitethatiscurrentlyunderconstruction-availableonrequest• http://www.archie-west.ac.uk• http://hpchub.sites.sheffield.ac.uk/research-groups• https://www.nottingham.ac.uk/research/news.aspx• Yes-containedintheannualreport• http://www.ucl.ac.uk/research-it-services/rits-case-studiesmoreindevelopmentatpresent• Yes,buttheseareusuallypublishedbyourPRO,forexample:
http://www.bristol.ac.uk/news/2014/november/hendra-in-bats-and-humans.html• https://www.liv.ac.uk/research/news/• Inamannerofspeaking,againseeN8HPCwebsiteandannualreport.• http://www.sussex.ac.uk/research/
Whousesyourservice?
Q46DoyouprovideHPCservicesbeyondyourimmediateorganisation?
HEILarge/
Specialist Regional GrandTotal
HEIs 3 3 1 7
Other....... 3 3 1 7
DiRAC 2 4
6
GridPP 3 2
5
N8HPC 4 1
5
BiochemistryandLifeSciencesInformatics
1 3
4
ARCHER 3 1
4
FarrInstitute 3 1
4
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixF–Servicemanagement:Fullbreakdownofsurveydata 49
SES/CFI 2 1
3
JASMIN/CEMS
2
2
CancerResearchUK 1 1
2
GenomicsEngland 1 1
2
HPCWales 1
1 2
NationalOceanographicCentre 1 1
2
ResearchDataFacility 1 1
2
MidPlus 2
2
ARCHIE-WeSt
1 1
DiamondLightSource
1
1
ELIXIR
1
1
HPCMidlands
1 1
HartreeCentre
1
1
MedicalBioinformaticsInitiative
1
1
WellcomeTrustSangerInstitute
1
1
AdministrativeDataResearchCentres
TheGenomeAnalysisCentre
Q47Whatservicesdoyouprovidetothirdparties?
HEI Large/Specialist Regional GrandTotal
Computeandstorage 8 10 3 21
Supportandtraining 3 6 3 12
Dataservices
7 2 9
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixF–Servicemanagement:Fullbreakdownofsurveydata 50
Networking
7 2 9
Softwaredevelopment 1 4 1 6
Businessdevelopment 1 2 2 5
Other........ 1
1
Q48HowmanyHEIsresearchgroupsuseyourservices?
Q49Publicandthirdsectororganisations
HEI Large/Specialist Regional GrandTotal
NHSand/orDepartmentofHealth
3 6 2 11
Charitablesector
6 1 7
DepartmentforBusiness,InnovationandSkills
1 5
6
EuropeanCommission
2
2
EuropeanSpaceAgency
1 1 2
Other......... 1 1
2
DepartmentofEnergyandClimateChange
0
0.5
1
1.5
2
2.5
3
3.5
0 20 40 60 80 100 120
Numb
erofres
pondents
Q48HowmanyHEIs'researchgroupsuseyourservices?
HEI
Large/Specialist
Regional
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixF–Servicemanagement:Fullbreakdownofsurveydata 51
Q50Whatpercentageofyourusersarefromindustry?
Q51Ofindustryusers,whatproportionareSMEs?
Q52WhatpercentageofsystemtimeisbeingusedbySMEs?Percentageofsystemcomputeresourceoveratwelvemonthperiod
Q53AreyoutakinganyspecificstepstoincreaseSMEuptake?
• Yes;advertisementandpersonalapproach.• Industryprogramme
0
50
100
150
Q50.Percentageofindustrialusers
Large/specialist
Regional
HEI
0
50
100
150
Q51.PercentageofindustrialuserswhoareSMEs
Large/specialist
Regional
HEI
0
5
10
15
20
25
Q52.PercentageofsystemomeusedbySMEs
Large/specialist
Regional
HEI
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixF–Servicemanagement:Fullbreakdownofsurveydata 52
• HavealwaysbeenontheforefrontofengagingSMEstouseHPCandcontinuetoworkextensivelyinthisarea.• Yes.ThisistheHartreeCentremission.WehaveaBusinessdevelopmentteamof5peopledevotedtothis.• TheHartreeCentreisdrivingSMEuptakeofHPCandwillprovideaseparateresponsetothissurvey.• No,focusinthesectorismoreonlargerPharma.• InstituteisplanningaBiodataInnovationCentreforupto30SME's• HPCWalesStage1iscurrentlycomingtoaclose,withthedirectionoftravelforitssuccessorprojectyettobe
quantified.WhileSMEengagementwillremainofimportanceitislikely,basedoninitialdiscussionswithWEFOandtheWG,thattheprofilewillbroaden
• Yes,dedicatedbusinessdevelopmentmanagerlookingtoworkwithSMEs• HoldingknowledgeexchangeeventsandexhibitingatindustryeventssuchasNAFEMS.Wehavealsoundertakena
securityauditofourfacilitybyanISOcompliantorganisation.WearealsointheprocessofattemptingtosecureinternalfundsforafulltimeBusinessDevelopmentrole
• Workwithbusinessengagementteams,theAdvancedManufacturingResearchCentreandAdvancedComputingResearchCentre.
• AsapartnerinHPCWales,wearedevelopingourindustrialcontactsthroughthismechanism.• TheCOREinitiativewithImperialCollegebringsthesepeopleinfromtimetotime.• OurstrategyforSMEinvolvementmirrorsourembeddeduseofinstitutionalBEteamswhichwecoordinateinorder
tosupportresearchthatusesHPCtoengagewithindustry.Broadly,ourapproachistotargetsupplychainsoflargercompanies.
• IncreasingengagementwithUniversityconsultingservicestobroadenoutreachofHPCactivity.• LiaisonwithSussexInnovationCentre
Q54Whatsectorsarebeingservedbyyourindustrialusers?
HEI Large/Specialist Regional GrandTotal
Advancedmaterialsandmanufacturing
7 1 3 11
Energyandenvironment 6 1 3 10
Lifesciences 3 4 2 9
Financeandprofessionalservices 3 1 2 6
Defenceandsecurity 4 1 1 6
Transport 3 1 1 5
Creativeindustries 1 1 2 4
Construction 2 1 1 4
Other..........
1 1 2
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixF–Servicemanagement:Fullbreakdownofsurveydata 53
Furtherinformation
Q55Pleaseusethisspaceforanyfurtherinformationyouwouldliketoprovide
• Aswiththesystemsurvey,thishasproveddifficulttocompletefromtheperspectiveofadistributedorganisationthatisprovidingHPCservicestobothacademicandthirdpartyorganisations.2.Theabilitytosaveandrestartthesurveywouldhavebeenhelpful.
• SoftwarelicensingstillamajorissueforSME's,bothintermsofprovidingabarriertoaccessandintermsofactingasaconstraintontheextenttowhichtheycanleveragethepowerofHPC.Consequently,thegainswearemakingwithSME'sarenotreflectedinthereport.
• Giventhelengthofthesurveyitwouldhavebeenhelpfultohavea"savenowandcompletelateroption"-plusbeingabletoprintormakecopiesofthecompletedsurveyreportforourownrecords(helpsfutureresponses!)
• Thesurveyquestionsareoftenunclear.
• ThereisactiveindustrialworkgoingonwiththeseparateVEC(VirtualEngineeringCentre)inpartnershipwithHartree.AllindustrialuseofLiverpoolHPCfacilitiesareviacollaboratingacademics.
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixG–Hardware:Thesurveyquestions 54
AppendixG–Hardware:Thesurveyquestions
Q1OrganizationnameQ2OrganizationalunitQ3EmailaddressQ4JobtitleQ5SystemnameQ6ExternalIPaddressofFQDNQ7URLforthewebsiteofthesystemoroverallserviceQ8Whatarethetopthreeresearchareasthesystemisusedfor?
Hardwarespecifications
Q9TotalnumberofprocessorcoresinthesystemQ10NumberofcomputenodesQ11NumberofprocessorcorespercomputenodeQ12RAMpercore(Gigabytes)Q13Computenodeprocessorspecificatione.g.IntelIvyBridgeE5-2670v22.5GHzQ14HowmanyGPUequippednodesdoesthesystemhave?Q15HowmanyXeonPhiequippednodesdoesthesystemhave?Q16Howmany"fat"nodesdoesthesystemhave,i.e.>=8GBRAMpercoreQ17DoesthesystemhaveadedicatedVisualizationcapability?PleasedescribebelowifapplicableQ18InterconnectSwitchFabrice.g.QDR/FDRInfiniBand,GigabitEthernet,NUMAlinkQ19Whenwasthesystemcommissioned?Q20Whenwillmaintenanceforthesystemterminate?
Storage
Q21Describethestoragecomponentofthesysteme.g.NetAppCDE5400,PanasasActivStor11,DirectAttachedStorageoncomputenodesQ22TotalusablestorageforHPCusersQ23Whatfilesystem(s)and/orobjectstoresdoyouuseforsharedstorage?Q24DoyousplitsystemstorageintermsofTBbetweenfast,tertiary,archivestorage?Q25Numberofregisteredusers
Performanceandconnectivity
Q26TheoreticalPeakPerformance(Tflop/s)Q27NodetoNodeDataRate(Gbit/s)Q28Averagenodetonodelatency(Microseconds)
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixG–Hardware:Thesurveyquestions 55
Q29TypicalCPUloadasa%ofoverallsystemQ30PeakinboundsustaineddatatransferratesQ31PeakoutboundsustaineddatatransferratesQ32IsthebandwidthabovededicatedforHPCserviceuse?Q33Specialconnectivityrequirementse.g.Lightpath,dedicatedcircuit,lowlatency
Softwareandoperatingenvironment
Q34InstalledsoftwareQ35WhatistheprimaryOperatingSystemyouuseoncomputenodes?Q36WhatistheprimaryOperatingSystemyouuseonhead/loginnodes?Q37Whatscheduler(s)doyouuse?Q38DoyouprovideaWebPortaltoyourusers?Ifso,pleasedescribebelowQ39DoyoubackupHPCuserdata?Q40Doyoudohavescheduledmaintenanceandifsohowoften?
Access,authorization,accountingandidentities
Q41ManagementofHPCsystemusersandprojectsQ42Accountingandresourceallocation
Furtherinformation
Q43Pleaseusethisspaceifthereisanyotherinformationyouwouldlikeshareaboutthesystem
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixH–Hardware:Summaryofthesurveydata 56
AppendixH–Hardware:Summaryofthesurveydata
Scienceareas-ofthe75systemsthatrespondentsreportedon,keyapplicationareaswereComputationalFluidDynamics(24%)materialsscience(16%)lifesciencesandbioinformatics(37%)andphysics(41%).
Hardware-withthegrowthinprocessorcorecounts,manyinstitutionalsystemswerenowonabar(incorecount,butnotnecessarilyincapacity)withanumberoftheregionalandlarge/specialistfacilities.SeveralHEIsalsohadmorecomputenodesthantheregionalcentresandanumberofthelarge/specialistfacilities.Thiscouldbeexpectedtochangewiththenextroundofhardwarerefresh.39%ofthesystemsinthesurveynowhadGPUequippednodes,althoughinmostcasesthesewere20nodesorlessoutofawholecluster.19%ofthesystemsinthesurveyhadXeonPhiequippednodes,butvirtuallyallofthesewere4orlessnodesoutofacluster.
Fat(highmemory)nodes-thevastmajorityofHEIandregionalsystemsstillonlyhaveahandfulof"fat"nodes,with8GBormoreofmemorypercore.Systemswithhundredsofhighmemorynodesarestillthedomainofthelargeandspecialistfacilities.
Visualization-37%ofrespondents'systemshadadedicatedvisualizationcapability.
Interconnect-48%ofrespondents'systemsusedQDRInfiniband,withagrandtotalof65%ofallsystemsinthesurveybeingconnectedviaInfiniband.23%ofsystemsinthesurveywereconnectedwithGigabitEthernet,with13%operatingat10Gbit/s,and2systemsoperatingat40Gbit/s.
Lifecycle-manyrespondentswerestillrunningoff-maintenancesystemscommissionedasfarbackas2007and2009.8newsystemshadbeencommissionedin2014,andafurther6in2015.26systems(35%oftheUKNeI)wouldfalloffmaintenancein2015and2016.
Storage-institutionsarenowstartingtoinchtowardsthesortofPetabytescalestoragesolutionswhichhadbeenverymuchthedomainoflargeandspecialistfacilities,with5HEIshavingstoragefacilitiesof1PBorabovefortheirscientificcomputingoperation.Lustre(33%),NFS(25%)andGPFS(18%)arestillbyfarthemostcommonfilesystems,althoughgrowthinPanasasinstallationshasseenPanFSriseto14%ofsurveyrespondents'systems.
Userbase-28%ofthesystemsinthesurveyhavelessthan100users,withonly16%havingmorethan1000users.
Performance-aswithcorecounts,institutionalsystemshavestartedtocatchupwithtier1and2facilities,with21outof75(28%)ofsystemsinthesurveyrunningat100Teraflops/sorbetter.However,notallrespondentsprovidedperformancestatsfortheirsystems.
Utilization-51%ofsystemswererunningat75%loadorabove,includingoversubscription,with75%generatingpeakinbounddataratesofbetween1and10Gbit/s.19%ofthesystemsinthesurveyweregeneratingover10Gbit/soutboundtraffic.30%ofsystemshadbeenprovidedwithdedicatedbandwidth,eitherintheirownright,orsharedwithother
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixH–Hardware:Summaryofthesurveydata 57
equipment.
Operatingenvironment-59%ofsystemsusedanunencumberedLinuxdistribution(suchasDebianorCentOS)astheirprimarycomputenodeoperatingsystem,with49%alsousinganunencumbereddistributionfortheirloginnodes.Bycontrast,fullysupporteddistributionssuchasRedHatEnterpriseLinux(RHEL)werefoundon24%ofloginnodesand15%ofcomputenodes.OnlyonerespondentwasrunningWindowsHPCServer.
Scheduler-SunGridEngineoritsderivativescontinuetodominatetheschedulersinthesurveyresponses,runningon24%ofsystems.However,SLURMhasnowgrownto16%oftheinstalledbase,with12installationsacrosstheUKNeI.
Backups-32%ofsystemsarenotbackedupatall,versus30%thatarebackedup,and37%wheresomedata(notnecessarilyincludinguserfiles)isbackedup.
Scheduledmaintenance-35%ofsystemsinthesurveydonothavescheduledmaintenance.43%ofsystemshaveeithermonthly,quarterlyorannualscheduledmaintenance.
Managementofusersandprojects-41%ofUKNeIsystemshaveauthenticationlinkedtoinstitutionalsystemssuchasActiveDirectory.However,44%ofsystemshaveatleastsomemanuallycuratedaccounts.32%ofsystemshaveapeerreviewprocessfornewprojects,althoughjust19%haveanequivalentprocessfornewuseraccounts.Only9%ofrespondentsautomaticallydeleteexpireduseraccounts,andjust11%updateuseraccountsautomaticallytotakeaccountofchangessuchasmovingdepartments.13%ofrespondentswereinterestedintriallingJisc'sAssentservice(formerlyProjectMoonshot).
Accountingandresourceallocation-51%ofrespondentsreportedthatprojectsontheirsystemsweregivenaresourceallocation,however23%statedthattheydidnotimposeanyresourcelimitsontheirusers.16%(12institutions)reportedthatusageoftheirsystemswasoftenconstrainedbysoftwarelicenses.
NeISurveyReport-2015
ProjectDirectorsGroup(PDG)
AppendixI–Hardware:Fullbreakdownofsurveydata
Q8Whatarethetopthreeresearchareasthesystemisusedfor?
TopresearchareasNumberofresponses
AdaptiveSystems 1
AeronauticalEngineering 1
AerospaceEng 1
Astronomy 3
Astrophysics 6
Atomicstructure 2
BigDataandDataAnalytics 2
Biochemistry 1
Bioinformatics 4
BiologicalSciences 4
BiomedicalSciences 2
CFD 18
CancerResearch 1
ChemicalEngineering 1
Chemistry 8
CivilandEnvironmentalEngineering 2
Climate/OceanModelling 4
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixI–Hardware:Fullbreakdownofsurveydata 59
ComputationalChemistry 4
ComputationalMedicine 1
ComputationalNeuroscience 1
ComputationalPhysics 1
ComputerAidedFormulation 2
CondensedMatter 2
Cosmology 5
Cryo-electronmicroscopy 1
EarthScience 4
Economics 1
ElectronicSystemDesign 1
EnergyEfficientComputing 1
EnergyEfficientTransport 1
Engineering 7
EnvironmentalGenomics 1
Exoplanets 1
FEA 2
FundamentalPhysics 1
GaitAnalysis 1
GalaxyFormation 2
Genomics 4
GeographicalSciences 1
Healthcare 2
HighEnergyPhysics 1
Hydrology 1
Informatics 2
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixI–Hardware:Fullbreakdownofsurveydata 60
LifeSciences 6
MHD 2
MachineLearning 1
MaterialsScience 12
Mathematics 5
MechanicalEngineering 1
MedicalInformatics 1
Microbialbioinformatics 1
MolecularDynamics 3
NaturalLanguage 1
Nextgenerationsequencing 1
Optoelectronics 1
ParticlePhysics 5
Physics 6
PlasmaPhysics 3
Pyschology 1
QCD 1
RNAsequencing 1
Satelliteimages 1
Science 1
SemiconductorDeviceModelling 2
SoftMatterPhysics 1
Speechandimageprocessing 1
Weather/climate 1
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixI–Hardware:Fullbreakdownofsurveydata 61
Hardwarespecifications
Q9Totalnumberofprocessorcoresinthesystem
Q10Numberofcomputenodes
Q11Numberofprocessorcorespercomputenode
0
50000
100000
150000
Q9Totalnumberofprocessorcoresinthesystem
Large/Specialist
Regional
HEI
0
1000
2000
3000
4000
5000
6000
7000
Q10Numberofcomputenodes
Large/Specialist
Regional
HEI
0500100015002000250030003500
Q11Numberofprocessorcorespercomputenode
Large/Specialist
Regional
HEI
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixI–Hardware:Fullbreakdownofsurveydata 62
Q12RAMpercore(Gigabytes)
Q13Computenodeprocessorspecificatione.g.IntelIvyBridgeE5-2670v22.5GHz
• 16xE55302.40GHz32xX56502.67GHz2xAMD61742.2GHz30xE5-2650v22.6GHz32xAMD63782.4GHz• 220*E5520,156*E5-2650v2• AMDOpteron62762.3GHz• AMDOpteron63762.3GHz• AMDOpteronProcessor63782.4GHz• E5-2580v2• E5-4620v22.6GHz• E54622.8GHz,E5-26702.6GHz• IBMPower• [email protected]• Intel• [email protected]• IntelE52650V2• IntelE5-26702.60GHz• IntelE5-4650L2.6GHz8coreXeonprocessor• IntelE5620• IntelHaswellE5-2640v32.6GHz• IntelIvyBridgeE5-26502.6GHz• IntelIvyBridgeE5-2650v22.6GHz• IntelSandyBridgeE5-26402.50GHz• IntelSandyBridgeE5-2650• IntelSandyBridgeE5-2660• IntelSandyBridgeE5-26702.6GHz,IntelWestmereX56602.8GHz,IntelHaswellE5-2680V32.5GHz• [email protected]• IntelSandybridgeE5-16503.2GHz• IntelSandybridgeE5-26502.0GHz• IntelSandybridgeE5-26702.6GHz• IntelWestmereE5-2697X56502.67GHz;IntelSandyBridgeE5-26902.9GHz;IntelSandyBridgeE5-26702.6GHz;• IntelWestmereX5650• IntelXeonE5-26602.2GHz• IntelXeonE5-2695v2• IntelXeonE55302.40GHz• IntelXeonX56602.80GHz
0
20
40
60
80
100
120
140
Q12RAMpercore
Large/Specialist
Regional
HEI
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixI–Hardware:Fullbreakdownofsurveydata 63
• Intel(R)Xeon(R)[email protected]• Intel(R)Xeon(R)[email protected]• Intel(R)Xeon(R)[email protected]• Intel(R)Xeon(R)[email protected](SandyBridge)• Intel(R)Xeon(R)[email protected]• Intel(R)Xeon(R)[email protected]• Intel(R)Xeon(R)[email protected]• IvyBridgeUoB:E5-4620v22.6Ghz,E7-8857v23.0Ghz,E7-8880v22.5Ghzothersites:E5-4610v22.3Ghz,E7-8850-v2
2.3Ghz• Ivybridge-v2E5-2650v22.6GHz• MixofWoodcrest,Nehalem,IvyBridge(variousmodelsthereof)• N/A• NA• [email protected],[email protected]• SCARFhasmanygenerationsofprocessorasthemodelistoupgradeityearly• SandyBridge,Westmere,Haswell,Opteron• [email protected]• Westmere2.8/2.95GHz• XeonE5-4620v2(20MCache,2.60GHz)• Xeon(R)CPUE5-26702.60GHz(3328cores)for"regular"computenode• latestintelgeneration• varies• variousgenerations• variousgenerationslatestinstall2015• westmere/ivybridge
Q14HowmanyGPUequippednodesdoesthesystemhave?
0
50
100
150
Q14.HowmanyGPUequippednodesdoesthesystemhave?
Large/Specialist
Regional
HEI
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixI–Hardware:Fullbreakdownofsurveydata 64
Q15HowmanyXeonPhiequippednodesdoesthesystemhave?
Q16Howmany"fat"nodesdoesthesystemhave,i.e.>=8GBRAMpercore
Q17DoesthesystemhaveadedicatedVisualizationcapability?Pleasedescribebelowifapplicable
0
10
20
30
40
50
Q15.HowmanyXeonPhiequippednodesdoesthesystemhave?
Large/Specialist
Regional
HEI
0100200300400500600700
Q16Howmany"fat"nodesdoesthesystemhave?
Large/Specialist
Regional
HEI
No63%
Yes37%
Q17.DoesthesystemhaveadedicatedVisualizaooncapability?
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixI–Hardware:Fullbreakdownofsurveydata 65
Q18InterconnectSwitchFabrice.g.QDR/FDRInfiniBand,GigabitEthernet,NUMAlink
Q19Whenwasthesystemcommissioned?
Q20Whenwillmaintenanceforthesystemterminate?
10GbE13%
1GbE7%
40GbE3%
BG/Q5DTorus2%
CrayAries1%FDR
Infiniband10%
IBM3%Infiniband
7%
QDRInfiniband
48%
SGINUMAlink
6%
Q18.Interconnect
2 2
6 6
13
8 8
6
0
2
4
6
8
10
12
14
2007 2009 2010 2011 2012 2013 2014 2015
Q19.Datecommissioned
21
13 13
7
43
4
0
2
4
6
8
10
12
14
2013 2014 2015 2016 2017 2018 2019 2020
Q20.Endofmaintenance
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixI–Hardware:Fullbreakdownofsurveydata 66
Storage
Q21Describethestoragecomponentofthesysteme.g.NetAppCDE5400,PanasasActivStor11,DirectAttachedStorageoncomputenodes
• 150TBLustre• 22TBScratchusingSGIStorageDirectlyAttached1xPanasasActivStor11• 250TBlustrescratch0.51PTIsilonresilientstorage• 2xSL8500taperobots(1usedbyTier1)with10000slotseachwithatotalnearlinecapabilityof160PB(whenfully
populated)TapeServers,OracleDatabases,DataTransferNodessupportingCASTORandDMFfrontendedbya700TBdiskcacheMonitoringSystems
• 7shelvesofPanasasActiveStor11• 800TBytesLustreDellPowerVault200TBytesNFSPowerVault• 8xHPP2000G3SASdisktrays• BlockdeviceaccessibleviaVMinuserconfiguredway• Cloud:Isilon&SAN,Cluster:GPFS• DDN• DDN9550,Panasas14s• DDNSFA10K• DDNSFA12K-40• DellMDstorage.• DellMD3000• Dellcommodity• DirectAttachonServer• DirectAttachedStorage• DirectAttachedStorageoncomputenodes• DirectAttachedStorageonstoragenodes• DirectAttachedstoragethroughheadnode• Directattached14PBofcommoditystorageusing400diskservers.SL85000taperobotwith>24driveswith14PB
managedbyCASTOR• Directattachedstorage• Directattachedstorageonnodes(0.5TBpernode)DDNSFA10K/SFA12KHPSL4540• EMCIsilon• Eachsitehas~400TBGPFSattachedtohypervisornodesforVMimagesandworkingdata.6.9PbofCEPHobjectstore
acrossthesitesisbeingcommissionedstill• Fast-PanasasActiveStor12Tertiary-HNASAMS2500&HUS150• Fraunhofer/BeeGFSparallelfilesystemoverIBusingDellMD3460• GPFS• GPFSBaseduponDDNdiscsviaInfiniband• GPFSBaseduponDDNdiscsviaInfiniband.GPFSbaseduponIBMGSS24• GPFSFPObaseduponIBMGSS24• GPFSacrossclusterwith250GBlocalstoragepernode• GPFSbaseduponIBMGSS24andGPFSbaseduponflashmemorydirectlyconnectedtoI/ONodes• IBMDS3512,DS3524andIBMDCS3700
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixI–Hardware:Fullbreakdownofsurveydata 67
• IBMDS5300• IBMGSS24+GSS26• IBMgeneralparallelfilesystem(GPFS),givingaround~110TBofusablestoragedividedamongtheusersandsplit
betweenpermanentstorageandtemporary/scratchspace.• LSI• LSISAS• LongTermNAS:HPSL4540basedNFSstoragetotalling696TBusable,splitacrosstwositesfordisasterrecoveryHigh-
speedscratch:IntelEnterpriseEditionLustreonSupermicrobasedhardwareprovidedbyBoston/BIOS-IT,2filesystemseachwith512TBusable
• Lustre120TB• LustreNFS• LustreParallelfilesystem,NFShomefilesystem,Nexenta• LustrescratchpartitionandbackedupNFShome• Netapp176TBusable(fordataandhomeareas)andlustre260TB(fastdataareawith90daylifetimepolicy)• PanasFSandFhGFS• PanasasActivStor11• PanasasActivStor11GPFS• PanasasActivStor12• PanasasActivStor14• PanasasActiveStor• PanasasActiveStor12• PanasasActiveStor14• PanasasActiveStor8(beingupgradedto16thisyear)forgeneraluseMultipleNFSdiskstoragenodesforHighEnergy
Physics• RAID5arraywithinheadnode• SGICXFS• SGIIS5000• SGIInfiniteStorage5000RAIDArrayIS5000dualcontroller2U24-bay2.5”RAIDarraycontaining:o20x3000GB7.2K
rpm2.5”6Gb/sSASHDD• StoragearraycomprisingDellMD3200andMD1200unitswithRAID6arrays• StorageisNFSmountedfromasinglestoragenode.• Sun/OracleSnowbirdsystemmadeupof8J4440arrays• TwoNetAppE2600• VariousDellMDstorage.
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixI–Hardware:Fullbreakdownofsurveydata 68
Q22TotalusablestorageforHPCusers
Q23Whatfilesystem(s)and/orobjectstoresdoyouuseforsharedstorage?
Q24DoyousplitsystemstorageintermsofTBbetweenfast,tertiary,archivestorage?
0
10000
20000
30000
TB
Q22.Totalusablestorageforusers
Large/Specialist
Regional
HEI
1
18
25
33
14
0
5
10
15
20
25
30
35
Ceph GPFS Lustre NFS PanFS
Q23.Filesystem/objectstore
38
21
9
Fast Ter|ary Archive0
5
10
15
20
25
30
35
40
Q24.Storagesplit
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixI–Hardware:Fullbreakdownofsurveydata 69
Q25Numberofregisteredusers
Performanceandconnectivity
Q26TheoreticalPeakPerformance(Tflop/s)
Q27NodetoNodeDataRate(Gbit/s)
0-10028%
100-20018%200-500
22%500-750
7%
750-10009%
1000-20008%
2000-50007%
Over100001%
Q25.Numberofregisteredusers
0
500
1000
1500
2000
2500
3000
Tflop/s
Q26.TheoreocalPeakPerformance
Large/Specialist
Regional
HEI
0
10
20
30
40
50
60
Gbit/s
Q27.Nodetonodedatarate
Large/Specialist
Regional
HEI
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixI–Hardware:Fullbreakdownofsurveydata 70
Q28Averagenodetonodelatency(Microseconds)
Q29TypicalCPUloadasa%ofoverallsystem
Q30Peakinboundsustaineddatatransferrates
0
20
40
60
80
100
120
Milliseconds
Q28.Averagenodetonodelatency
Large/Specialist
Regional
HEI
Commissioning5%
<=25%6%
26%to50%6%
51%to75%32%
>75%45%
Oversubscribed6%
Q29.TypicalCPUloadasa%ofoverallsystem
100Mbit/sto1Gbit/s
2%
1Gbit/sto10Gbit/s75%
Other4%
Unsure15%
Q30.Peakinbounddatarates
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixI–Hardware:Fullbreakdownofsurveydata 71
Q31Peakoutboundsustaineddatatransferrates.Inmostcasesthiswillbetheconnectionspeedoftheorganizationoverall,butifbandwidthisspecificallydedicatedforHPCpleaseindicatebelow
Q32IsthebandwidthabovededicatedforHPCserviceuse?
Q33Specialconnectivityrequirementse.g.Lightpath,dedicatedcircuit,lowlatency
• Replicatedstoragebetweensitesmayinthefuturerequirededicatedconnectivity.Wecurrentlyareprovisioningipsectunnelsbetweensitesforsecurereplicationofdata.
• Canconfigurelightpathordedicatedconnectionsasrequired.• 10GbexternallinktoJanet,40GBinternalbackplane.• OPNtoCERNat10Gbits/s.shared40Gbits/stoJanet.• TheNSCCSmachineisalargesharedmemorymachinewith512Coresand4TBofmemory.Thisarchitecturesuitsthe
computationalchemistryapplications.• OPNstoMetOfficeExeter,ARCHER,LeedsUniversity,SpaceApplicationsCatapult.• Lightpathanddedicatednetworkscanbeconfiguredonrequest.• AfunctionofApplicationSector-directconnectivitytoNGSsystemsdemandspeakperformance.• Noneatpresent,butNGSaccessdictatesanorderofmagnitudeincreaseinconnectivityperformance.
<1Gbit/s9%
1Gbit/sto10Gbit/s55%
>10Gbit/s19%
Other4%
Unsure13%
Q31.Peakoutbounddatarates
Yes,singlesystem18%
Yes,mul|plesystems22%
No50%
Other4%
Unsure6%
Q32.Dedicatedbandwidth
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixI–Hardware:Fullbreakdownofsurveydata 72
Softwareandoperatingenvironment
Q34Installedsoftware
Theresultsfromthisquestionwereinconclusiveduetoabuginthesurveytool.
Q35WhatistheprimaryOperatingSystemyouuseoncomputenodes?
Q36WhatistheprimaryOperatingSystemyouuseonhead/loginnodes?
11 13
44
16
05
101520253035404550
Fullysupported1stpartyLinuxdistribu|on
HPCVendorre-spinofexisitngLinuxdistribu|on
UnencumberedLinuxdistribu|on
WindowsHPCServer
Other
Q35.PrimarycomputenodeOS
18
6
37
1
7
0
5
10
15
20
25
30
35
40
Fullysupported1stpartyLinuxdistribu|on
HPCVendorre-spinofexisitngLinuxdistribu|on
UnencumberedLinuxdistribu|on
WindowsHPCServer
Other
Q36.PrimaryloginnodeOS
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixI–Hardware:Fullbreakdownofsurveydata 73
Q37Whatscheduler(s)doyouuse?
Q38DoyouprovideaWebPortaltoyourusers?Ifso,pleasedescribebelow
• Horizoniscurrentlyonlywayformostuserstoaccess.APIaccessprovidedviacontrollers.• Onrequest• Yes,SAFE:https://www.archer.ac.uk/safeAccountManagementProjectManagementReportingHelpdeskIncident
ManagementSystemConfiguration• http://www.cosmos.damtp.cam.ac.uk/• Yes-OpenStackHorizon• AVFisaphysicalfacilitywhereuserscomeandexploretheirdataandinteractin3D.AVFcanalsoofferremote
visualisationaswell.• CertificateWizardApplicationhttp://www.ngs.ac.uk/ukca/certificates/certwizard• Yes,needsauthenticationtousehttps://goc.egi.eu/portal/• SCARFisaccessibleviathePlatformApplicationPortal.Thisallowsuserstouploadanddownloadfiles,submit
computationaljobsanddesignworkflows.• WebaccessviathePlatformApplicationPortal• Userprovisionedcloudwebportalavailable.• ThewebportaltotheaccountingdataisprovidedbyCESGAhttp://accounting.egi.eu/egi.php• Yes,basedonFujitsu'sSynfiniWaymiddleware;seehttps://portal.hpcwales.co.uk• Yes,foraccessdetailsandusageinformation.Notforjobsubmission.• http://www.cfi.ses.ac.uk/cfi/iridis/• Yes-Galaxy(LifeSciences),WebMO(Chemistry),GridChem(Chemistry)andweareintheprocessofevaluatingboth
Altair'sComputeManagerandOpen-SourceCylc(asamulti-purposeweb-basedinterface).• Forgaussianonly• EvaluatingAltairComputeManager,notinproductionuse• Oraclesecureglobaldesktophttp://www.oracle.com/us/technologies/virtualization/secure-global-
desktop/overview/index.html• https://maxwell.abdn.ac.uk-providedbyAlces• StandardBrightClusterManagerwithmodificationsbyClusterVisionforVisualisationservice
13
6
14
912
18
8
0
5
10
15
20
Q37.Scheduler
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixI–Hardware:Fullbreakdownofsurveydata 74
Q39DoyoubackupHPCuserdata?
Q40Doyoudohavescheduledmaintenanceandifsohowoften?
Access,authorization,accountingandidentities
Q41ManagementofHPCsystemusersandprojects
Yes30%
Somedata37%Onlyhome
dirs1%
No32%
Q39.Backups
Monthly22%
Quarterly10%
Annually11%
Other22%
No35%
Q40.Scheduledmaintenance
8
31
10
21
7 74
10
33
4
14
24
05101520253035
Q41.Managementofusers&projects
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixI–Hardware:Fullbreakdownofsurveydata 75
Q42Accountingandresourceallocation
Furtherinformation
Q43Pleaseusethisspaceifthereisanyotherinformationyouwouldlikeshareaboutthesystem
• CLIMBisadifferentapproachcomparedtotraditionalHPCsystems,somanyofthequestionsdon’tquitefit!StoragecephnodesalsorunRHEL7.1
• TheComputingInfrastructureforScience(CiS)groupinNBIParnershipLtdmanagestheHPCandenterprisestorageforfourInstitutesonasharedcampusnetworkinNorwich:TGAC,JIC,IFRandTSL.ThelargestproportionofHPCandstorageutilisationisfromTGAC.
• TheAVFisaphysicalfacilitywithhighperformance3Dvisualisationcapabilityattachedtohighmemorymachinesfordataanalysis.Theroomisusedcollaborativelywithresearchersexploringtheirdatatogetherandgainingnewinsights.Thehighmemorynodesarecriticalsuchthatlargedatasetscanbemanipulatedinrealtime.Remotevisualisationisalsoofferedtoresearchers
• PerformanceismeasuredbytheHEPSPECratherthaninTflopsasthisismorerelevanttotheworkloads.http://w3.hepix.org/benchmarks/doku.php?id=homepageTheperformanceoftheTier1is~120kHEPSPEC06
• TheAtlasDatastoreisahierarchicalstoragesystemprovidingarchive,backups,datacurationandworkingrepositories.
• TheUKe-ScienceCertificateAuthorityprovidesasecurityinfrastructureforpeopleandsystemswhichisacceptedgloballythroughtheIGTF.TheIGTFistheinteroperableglobaltrustfederation.http://www.igtf.net
• GOCDBistheofficialrepositoryforstoringandpresentingEGItopologyandresourcesinformation.TheGOCDBdataconsistsmainlyof:ParticipatingNationalGridInitiatives(NGI)GridSitesprovidingresourcestotheinfrastructureResourcesandservices,includingmaintenanceplansfortheseresourcesParticipatingpeople,andtheirroleswithinEGIoperations
38
12
2523
17
13
0
5
10
15
20
25
30
35
40
Projectsaregivenaresource
alloca|on
Usageiso}enconstrainedby
so}warelicenses
Useraccountsaregivenaresourcealloca|on
Weac|velymonitorusageagainstresource
alloca|on
Wedonotimposeresource
limits
Other
Q42.Accounong&resourceallocaoon
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixI–Hardware:Fullbreakdownofsurveydata 76
• SCARFismanagedaspartofaportfolioofservicesthatSCDofferstotheUKandinternationalsciencecommunities.• NSCCSispartofancollectionofservicesthatSCDprovidestotheUKacademiccommunityOverthelastyearNSCCS
hasbeenusedby54researchgroupsfrom24institutionsresultinginover60publications.• Data-intensivecomputingJASMINprovidestheUKandEuropeanNERCfundedenvironmentalsciencecommunities
withanefficientdataanalysisenvironment.Manydatasets,particularlymodeldata,aretoobigtobeeasilyshippedaround:JASMINenablesscientiststobringtheirprocessingtothedata.FlexibledataaccessJASMINprovidesnewwaysforscientiststocollaborateinself-managinggroupworkspaces,enablingmodelsandalgorithmstobeevaluatedalongsidecuratedarchivedata,andfordatatobesharedandevaluatedbeforebeingdepositedinthepermanentarchive.
• APELisanaccountingtoolthatcollectsaccountingdatafromsitesparticipatingintheEGIandWLCGinfrastructuresaswellasfromsitesbelongingtootherGridorganisationsthatarecollaboratingwithEGI,includingOSG,NorduGridandINFN.TheaccountinginformationisgatheredfromdifferentsensorsintoacentralaccountingdatabasewhereitisprocessedtogeneratestatisticalsummariesthatareavailablethroughtheEGI/WLCGAccountingPortal.APELcollectsdatafrom~300institutionsat~3Mrecordsperdaytotalling~400GBofdata.Thisiskeptfor18monthsafterwhichsummarydataisheld.StatisticsareavailableforviewindifferentdetailbyUsers,VOManagers,SiteAdministratorsandanonymoususersaccordingtowelldefinedaccessrights.
• TheaccountinginformationisgatheredfromdifferentsensorsintoacentralaccountingdatabasewhereitisprocessedtogeneratestatisticalsummariesthatareavailablethroughtheEGI/WLCGAccountingPortal.TheAPELsystemreceives~3MrecordsperdayintoaMySQLdatabasefrom314sites.IndividualJobrecordsarekeptfor18months(~400GB).After18monthsthejobrecordsaresummarisedandkeptindefinitely.StatisticsareavailableforviewindifferentdetailbyUsers,VOManagers,SiteAdministratorsandanonymoususersaccordingtowelldefinedaccessrights.
• Backupisfordisasterrecoverypurposesonly• Acommenttopointoutthedifficultyinenteringdatafordistributedsystems-particularlytoughinourcasee.g.,Q19
-couldnotworkouthowtoentermultipledatesforamulti-phaseimplementation-2011and2013.Onethingtoredlineanon-acceptableresponse,butnoindicationofwhatconstitutesavalidresponse.Wouldhavebeenusefultohavehadasaveandrestartcapabilityforaquestionnaireofthissize.Curtailingtheinputonsomeofthelinesshouldbeaccompaniedbyamaximumfiledlength.
• Althoughlistedasasinglesystem,therearethreepartitions-theprimarySandyBridgepartitionisthemainparallelMPIservice,theWestmereisforserialorbatchjobs,andthenewHaswellservicewillhaveamixedworkload(primarilyserialbutwillalsobeavailableforMPI-basedjobs).ThislatterpartitionwasinstalledinJanuary2015andthemaintenancewillexpireinJan2018(butunfortunatelythemaintenancequestiononlypermittedasingledateentry!).
• Heterogeneoussystem,somultiplehardwaretypesandsupportcontractspanning2010todate.Surveydoesn'treallyaccommodatethistypeofcluster.TheclusterisbothourlocalHPCfacilityandHEP/GridPPnode.Fundingisahybridmodel-acentralUniversitycontributionandresearchgrant-fundedcontributions
• Researchershavetoregisteraproject,whichmaybeunfunded,andusershavetobeassociatedwithaproject.Thisenableslinkingresearchstudentstotheirsupervisorwhichisveryusefulincaseofproblems.ItalsoenablesustoidentifytherangeofusersanddisciplineswhichisessentialtomakingthecasetotheUniversityforongoingfunding.
• WeareabouttoundergoasoftwareupgradethatwillratherchangetheOSandapplicationavailability(forexamplewearemovingtoRHELfromSL).Commissiondateisinaccurateasthissystemhasbeeninuseforalongtime.Itisalsonotpossibletogiveamaintenancecutoffbecausethatvariesfromalreadyexpiredtobrandnewequipment.
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixI–Hardware:Fullbreakdownofsurveydata 77
• Iskippedthequestionsoncommissioningandmaintenancedates,becauseweupgradeonanannualcycle,andoncetheoriginalsupportagreementexpiresweoftenrolltheHPCnodeintoourcampusmaintenanceagreementwithHP.
• WealsohaveanactiveCondorsystemwithcirca20activeusers,runningacross750campusPCs,thattypicallyaccommodates20,000coredaysofcomputingpermonthviaapproximately60,000jobs(permonth).ThisrelievespressureofftheHPCsystemforhighvolume,highthroughoutjobs.
• Thesystemisheterogeneousandbasedonacontributionmodel.Centralseedfundingboughtinfrastructure.Researchgroupscontributefundstobuycomputenodes.Procurementstakeplacetwoorthreetimesayear.
• TheUniversityfundsageneralHPCservicewhichisaugmentedbytheadditionofresearchgrantorschoolfundedcomputeandstorage.NodesandStoragearepurchasedwith5yearswarranty,afterwhichtheyareconsideredEndofLife.WeuseBrightforClusterprovisioningandmanagement.
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixJ–Hardware:Summaryofthesurveydata 78
AppendixJ–Hardware:Summaryofthesurveydata
Table:ThehardwaresystemsaregivenintheTablebelowwhichissplittointo3sections,eachrepresentingalayerintheBranscombPyramid.Section1liststheLargeandSpecialistSystems.Section2givestheRegionalSystems.Section3liststhefoundationallayer,theHEIsector.
1.LargeandSpecialistServices
Organizationname Systemname
Whatarethetopthreeresearchareasthesystemisusedfor?
Totalnumberofprocessorcoresinthesystem
TotalusablestorageforHPCusers(TB)
Numberofregisteredusers
TheoreticalPeakPerformance(Tflop/s)
CloudInfrastructureforMicrobialBioinformatics-CLIMB-Birmingham,Cardiff,Swansea,Cardiff CLIMB
ResearchinhowmicrobialbioinformaticiansusecloudMicrobialbioinformatics(assembly,analysis)Buildinganacademiccloud 3,864
0-100 74
DiRAC@DurhamUniversity
DiRAC-1@Virgo(COSMA4)
CosmologyGalaxyFormation 7,072 1,100 100-200 32
DiRAC@DurhamUniversity
DiRAC-2@DataCentric(COSMA5)
CosmologyMHDGalaxyFormation 6,720 2,500 100-200 140
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixJ–Hardware:Summaryofthesurveydata 79
DiRAC@EPCC DIRACBG/QQCD,SoftMatterPhysics 98,304 1,000 200-500 1,258
DiRAC@UniversityofCambridge(DAMTP) COSMOS
Cosmology,Astrophysics,Exoplanets 3,284 306 200-500 63
DiRAC@UniversityofCambridge(HPCS) Darwin
LifeSciences.Atomicstructure.ComputationalFluidDynamics. 9,600 2,847
750-1,000 200
DiRAC@UniversityofLeicester Complexity
AstrophysicsParticlephysics 4,352 710 100-200 91
EMBL-EBI-EuropeanBioinformaticsInstitute
EmbassyCloud Lifescienceresearch 31,000 3,200 200-500
eMedLab-TheCrick,UCL,QMUL,LSHTM,EBI,TheSanger eMedLab
Biomedicalresearch(andanyassociatedareas).NextgenerationsequencingRNAsequencingCryo-electronmicroscopy 6,048 4,800 0-100
UVRI/MRCMedicalInformaticsCentre UMIC
MedicalInformatics(notyetinoperation) 2,048 1,720 0-100 19
NorwichBioscienceInstitutes(TGAC,JIC,IFR,TSL)
Bioinformatics,mathematicalmodelling. 9,000 4,000
750-1,000
STFCHartreeCentre BlueJoule
Modelling&Simulation(CFD,Materials,andComputerAidedFormulation) 98,000 6000 200-500 1,200
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixJ–Hardware:Summaryofthesurveydata 80
STFCHartreeCentre BGAS
Modelling&Simulation(CFDandMaterials) 32,000 1000 0-100 450
STFCHartreeCentre BlueWonder
Modelling&Simulation(CFD,Materials,andComputerAidedFormulation) 24,000 9000
750-1,000 200
STFCHartreeCentre EECR/FPGA
EnergyEfficientComputing 2,000 200 0-100
STFCHartreeCentre BigData
BigDataandDataAnalytics 1,184 1000 0-100
STFCScientificComputingDivision SCARF
ComputationalChemistryPlasmaPhysics,ProcessingSatelliteimagesSupportofISIS,CLF,RAPSP,DLSusercommunities 7,000 320 500-750 165
STFCScientificComputingDivision JASMIN
ClimateScience,EarthObservation,environmentalgenomics 4,500 25
Over10,000
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixJ–Hardware:Summaryofthesurveydata 81
STFCScientificComputingDivision
NationalServiceforComputationalChemistrySoftware
TheEPSRCUKNationalServiceforComputationalChemistrySoftware(NSCCS)providesaccesstosoftware,specialistconsultation,computingresourcesandsoftwaretrainingtosupportUKacademicsworkingacrossallfieldsofchemistry.NSCCSsupports127researchgroupsfromdisciplinesincludingChemistry,MaterialsSciences,Physics,EarthScienceandEngineering,Astronomy,Biochemistry,BiologicalSciences,BiomedicalSciences,LifeSciences,CivilandEnvironmentalEngineeringandChemicalEngineering 512 32 200-500
STFCScientificComputingDivision
AtlasVisualisationFacility(AVF)
MaterialsciencetomographyandPlasmaPhysics 112
0-100
STFCScientificComputingDivision
AtlasDatastore
ParticlePhysics,LifeSciences,MaterialsSciencepluseverythingelse
25,000 100-200
STFCScientificComputingDivision
UKe-ScienceCertficationAuthority
SupportsallUKresearch.MajorusersParticlePhysics
750-1,000
STFCScientificComputingDivision GOCDB
Supportsmulti-nationale-infrastructures
2,000-5,000
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixJ–Hardware:Summaryofthesurveydata 82
STFCScientificComputingDivision GridPPTier1
ParticlePhysicsExpandingtosupportotherdisciplines 10,000 28,000
1,000-2,000
STFCScientificComputingDivision APEL
APELisacentralrepositoryforaccounting/usagedata.Itsupportslocal,gridandcloudcomputing.
STFCScientificComputingDivision APEL
APELisanaccountingtoolthatcollectsaccountingdatafromsitesparticipatingintheEGIandWLCGinfrastructuresaswellasfromsitesbelongingtootherGridorganisationsthatarecollaboratingwithEGI,includingOSG,NorduGridandINFN.APELcurrentlysupportslocal,gridandclouddata
200-500
TheInstituteofCancerResearch Multiple
Processingofsequencing,massspecandimagingdata. 1,600 2,500 0-100
EPCC ARCHER
MaterialsScience,Climate/OceanModelling,ComputationalFluidDynamics 118,080 4,608
1,000-2,000 2,550
EPCC Indy Industry/SMEUse 1,536 175 100-200
EPCC UltraHealthcare,Bioinformatics 512 253 0-100
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixJ–Hardware:Summaryofthesurveydata 83
EPCC DIR Dataanalytics 240 2,250 0-100
EPCC UK-RDF
Climate/oceanmodelling,ComputationalFluidDynamics,MaterialsScience
23,0001,000-2,000
FarrNorth,HealtheResearchCentre
HeRCSafeHaven
HealthcareBio-healthInformaticsMachineLearning 256
0-100
WellcomeTrustSangerInstitute
SangerHPCResources Genomics 16,946 9,900
1,000-2,000
LARGE&SPECIALISTTOTALS 499,770 135,446 30,550 6,441
2.RegionalSystems
Organizationname Systemname
Whatarethetopthreeresearchareasthesystemisusedfor?
Totalnumberofprocessorcoresinthe
system
TotalusablestorageforHPC
users(TB)
Numberof
registeredusers
TheoreticalPeak
Performance(Tflop/s)
HighPerformanceComputing(HPC)Wales
Various(distributedsystem)
AdvancedMaterials&Manufacturing,LifeSciencesandEnergy&Environment 16,816 702
2,000-5,000 319
HPCMidlands Hera
AdvancedMaterialsEnergyEfficientTransport 3,008 120 100-200 48
N8HPC Polaris
5,312 175 200-500 138
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixJ–Hardware:Summaryofthesurveydata 84
ARCHIE-WeSt ARCHIEMoleculardynamics,CFD,PlasmaPhysics 3,920 148 200-500 38
SES/CFI IRIDIS3
Chemistryresearch,EngineeringresearchandBiologyresearch 12,000 110 200-500 106
REGIONALTOTALS 41,056 1,255 49,660 649
3.HEISystems
Organizationname Systemname
Whatarethetopthreeresearchareasthesystemisusedfor?
Totalnumberofprocessorcoresinthe
system
TotalusablestorageforHPC
users(TB)
Numberof
registeredusers
TheoreticalPeak
Performance(Tflop/s)
CardiffUniversity Raven
EPSRC(materials,chemistry,engineering),BBSRC(genomics)andNERC(earthsciences) 4,352 275 500-750 110
CranfieldUniversity Astral CFDFEA 1,280 34 200-500 20
DurhamUniversity Hamilton
CondensedMatterMolecularDynamicsFluidDynamics 5,600 350 200-500 75
ImperialCollegeLondon ax3 Genomics 1,300 1,500 0-100
ImperialCollegeLondon cx1
21,558 2,000750-1,000
ImperialCollegeLondon cx2
7,000 500 0-100 60
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixJ–Hardware:Summaryofthesurveydata 85
King'sCollegeLondon ADA
ComputationalPhysics,Mathematics,Informatics 1,624 87 100-200 85
LancasterUniversity
HEC(HighEndCluster)
HighEnergyPhysicsCondensedMatterTheoryCFD 4,784 1,530 200-500
LoughboroughUniversity Hydra
2,460 100 100-200
QueensUniversityofBelfast DellCluster
Chemistry,Physics,CancerResearch 984 40 0-100
QueensUniversityofBelfast
WindowsCluster
biology,speechandimageprocessing,CFD 256 30 0-100
TheUniversityofBirmingham
bluebear.bham.ac.uk
MathematicsCivilEngineeringPyschology 1,632 150 500-750 21
TheUniversityofNottingham Minerva
Chemistry,Engineering,Physics 2,752 180 200-500 55
TheUniversityofSheffield iceberg1
AeronauticalEngineeringandmodellingofturbulentfluidsComputationalMedicineBioinformatics 3,440 40
1,000-2,000 112
UniversityCollegeLondon Legion
Chemistry,Physics,BiologicalSciences(accordingtoREFCategories) 7,816 356 500-750 115
UniversityofAberdeen
maxwell.abdn.ac.uk
LifeSciences-GenomicsCFD-EngineeringMatlab 600 56 100-200
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixJ–Hardware:Summaryofthesurveydata 86
UniversityofBath
Balena/Aquila
ChemistryPhysicsMechanicalEngineering 3,072 220 0-100 64
UniversityofBristol BlueCrystal
Chemistry,AerospaceEng,GeographicalSciences 9,000 740
750-1,000 240
UniversityofCambridge Wilkes
ComputationalFluidDynamics.Atomicstructure. 1,536
0-100 256
UniversityofEdinburgh Eddie
Physics,Informatics,Engineering 3,248 281
1,000-2,000 28
UniversityofExeter
AstrophysicsWeather/climateHydrology 2,184 73 100-200 25
UniversityofGlasgow Conan
SemiconductorDeviceModelling 1,360 40 0-100
UniversityofGlasgow Cnoc
ElectronicSystemDesign 320 10 0-100
UniversityofGlasgow Dusty
ComputationalFluidDynamics 188 4 0-100
UniversityofGlasgow Miffy
SemiconductorDeviceModellingComputationalFluidDynamicsOptoelectronics 1,256 22 0-100 15
UniversityofLeeds Arc1
CFD,Astrophysics,climatescience 4,128 117 500-750 31
UniversityofLeeds Arc2
3,040 175 200-500 316
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixJ–Hardware:Summaryofthesurveydata 87
UniversityofLeicester ALICE
Astrophysics(nowlargelymigratedtoDiRAC)EarthObservationScienceEngineeringEconomics 3,972 1,644 200-500 101
UniversityofLiverpool chadwick
MaterialsmodellingComputationalFluidDynamicsGaitAnalysis 3,180 132 100-200 23
UniversityofManchester
ComputationalSharedFacility
ComputationalChemistry/MDCFDFEA 6,288 750
750-1,000 111
UniversityofOxford Arcus-A
1,728
2,000-5,000 55
UniversityofOxford Arcus-B
5,440 4322,000-5,000 538
UniversityofOxford Arcus-GPU
12
2,000-5,000 146
UniversityofPortsmouth Sciama
fundamentalphysics,cosmologyandastrophysics 3,704 740 100-200
UniversityofStAndrews wardlaw
MHD,Astronomy,Chemistry 3,510 150 200-500 33
UniversityofSussex Apollo
Physics(Astronomy,Cosmology,Particle),Engineering(CFD),Informatics(ComputationalNeuroscience,Adaptivesystems,NaturalLanguage) 3,248 560 100-200
HEITOTALS 127,852 13,318 30,430 2,635
NeiSurvey2015ProjectDirectorsGroup(PDG)
AppendixJ–Hardware:Summaryofthesurveydata 88
Totalnumberofprocessorcoresinthesystem
TotalusablestorageforHPC
users(TB)
Numberof
registeredusers
TheoreticalPeak
Performance(Tflop/s)
GRANDTOTALS 1,209,504 286,720 67,680 16,815
(upperbound)