# Lev Lafayette, 2020, Spartan: From Experimental Hybrid towards a Petascale Future, Presentation to EResearchAustralasia, October 20 https://conference.eresearch.edu.au/2020/09/spartan-from-experimental-hybrid-towards-a-petascale-future/ Spartan is the general purpose high performance computing HPC system at the University of Melbourne, first introduced in 2016. Prior to that general researcher access was provided by a cluster named "Edward", which itself was the the successor, in 2011, to a system called "Alfred"; the previous lead systems engineer adopting a naming convention based on the Kings of Wessex. Continuing that convention would have been difficult, as the next in succession was "Æthelstan" and/or "Ælfweard"; there is conflicting documentation on who took over the Kingdonm or whether it was divided [1]. In either case, it may have been difficult to enter such characters for a login. "Edward" was a fairly typical cluster; there was a login node, a management node, 2 storage servers and 48 nodes, and, experimentally, 2 GPU nodes. The operating system was Scientific Linux 6, it used Moab as the job scheduler, TORQUE as the resource manager, and xCAT for node deployment and management tool for user and project quotas. The management node also provided /usr/local to the entire cluster via NFS. The compute nodes would mount /home, and /project directories from the storage nodes. All of this is fairly typical of a small HPC system, and a general model that is well known in numerous institutions across the world. Like most institutions when a cluster is reaching end-of-life, the University of Melbourne asked the main researchers what they want from a new system, and their answers predictable; more nodes, faster processors, more memory, faster interconnect, more storage etc. But what people say they want and what they actually want can be two different things; especially when they are speaking from a position that is not fully-informed of the system and operations interact, and especially without reference to finances. As previously reported to eResearchAustralasia [2] an analysis of actual job metrics indicated that 73.35% of jobs in the final year were single core. This is not inefficient it itself, but rather a different and not-uncommon way of making use of an HPC system [3]. An HPC system can be used for many jobs, that is, capacity, or it can be used for large, multi-node jobs, that is, capability. But what doesn't make sense, from an opportunity cost point of view, is building a system with an expensive, high-speed interconnect throughout, when only a small proportion of the jobs will actually make use of such hardware. With a limited budget (the name "Spartan" was coined on account of our laconic finances) a decision was made to introduce an HPC system that was quite innovative in terms of many of the technologies, with an explicit emphasis on high throughput more than high performance. A "bare-metal" HPC with a high-speed RoCE interconnect was used for a smaller partition, where as a much larger "cloud" partition came from the existing from the National eResearch Collaboration Tools and Resources project (NeCTAR) OpenStack-based research cloud, which was also used for deployment. These virtual machines, it must be mentioned, were on a one-to-one ratio with physical hardware, as it was discovered quite early that that over-commit on cloud nodes was not viable due to the possibility of time-mismatch errors in a large proteomics experiment. The RoCE interconnect actually showed better performance in i/o operations than a comparable system using FDR14 Infiniband. Slurm Workload Manager was used for job scheduling and resource management, Easybuild and LMod for software builds and environment modules, a rather than a standard parallel file system such as Lustre or GPFS etc, Spartan made use of CephFS. Many of these choices were quite novel and interesting, and presentations concerning Spartan were subject to a bit of an international tour in 2016 with a starting with an early preview in New Zealand at Multicore World in February 2016, before eResearch in Australia, 11th October 2016, then the Center for Scientific Computing (CSC), Goethe University Frankfurt, on the 14th of October, High Performance Computing Center, University of Stuttgart on the 18th of October, High Performance Computing Centre, Albert-Ludwigs-University Freiburg on the 19th of October, European Organization for Nuclear Research (CERN) on the 20th of October, Centre Informatique National de l’Enseignement Supérieur, Montpellier on the 21st October; and the Centro Nacional de Supercomputación, Barcelona, on November 2nd, 2016. The main presentation however, was to the OpenStack Summit in Barcelona on October 27 [4][5]. The success of Spartan's architecture soon became apparent. Whilst Edward had completed just over 375,000 jobs in 2015, Spartan completed more than a million in its first year from launch. The system expanded with additional compute nodes from specialist projects, departments, and research agencies that had purchased their own hardware but saw the advantage of having software installs and system administration tasks managed centrally. But the most significant expansion was the addition of a substantial GPGPU partition, of 68 nodes and 272 nVidia P100 GPGPU cards, funded by Linkage Infrastructure, Equipment and Facilities (LIEF) grant LE170100200 from the Australian Research Council, in a partnership between the University of Melbourne, La Trobe University, Deakin University, and the Royal Melbourne Institute of Technology (RMIT). One particular use of the GPU partition that has been previously reported was a joint effort with Nyriad from New Zealand, which optimised MATLAB code for marine biodiversity population samples which had been previously run as job arrays [6]. During this period we also introduced FastX for interactive remote desktops, and interactive sessions through Open OnDemand for Jupyer notebooks and RStudio. Whilst the use of the NeCTAR cloud provided a great deal of flexibility for us in terms of deployment it is not virtual machines all the way down. We were fortunate, in terms of proximity, to have the traditional HPC system in close proximity to the cloud nodes. Many of the great promises or assumptions of "HPC in the Cloud" is, it must be said, are a bit of a myth and several attempts by vendors to sell us "HPC in the Cloud" solutions to our extend our HPC-cloud chimera did not develop far, invariably due to latency and distance issues and the relatively unique specific configurations that all HPC systems have. A replication of the existing HPCs environment would be required, including home or project directories, the Slurm controller daemon and database, LDAP, specialised application installations, etc. Indeed, the most successful argument for "HPC in the Cloud" really comes from Moab/NODUS cloudbursting by Adaptive Computing which notes its primary use for external cloud-bursting is business reasons (e.g., SLAs) rather than technical reasons. [7] All of this should be considered in the context of the University of Melbourne's Petascale Campus Initiative (PCI) which is a response to the need for data-intensive research. Operating for five years from 2018 to 2022, it aims to provide the infrastructure, the career path, data management, and disciplinary scope for the University. Whilst the scope of the PCI is too great to fully elaborate here, although much of the research activities can be found through the Melbourne Data Analytics Platform (MDAP) and their publication, Shaping the future of data-intensive research [8]. Suffice to say that it makes significant use of the Research Computing Services team which operates and manages Spartan along with, it must be noted, an extensive and extremely popular programme of HPC training workshops. As previously illustrated there is a very strong correlation between HPC training and HPC utilisation by researchers [9], that adds to the massive macroeconomic benefits that result from HPC systems - an estimated $44 in profits or cost-savings per $1 invested [10], primarily in externalities; the Solow residual in effect. In its six years of operation, from 2010 to 2016, the Edward HPC system ran a total of 1,434,474 jobs (not including array subjobs) from 886 user across 371 projects. With around four-and-a-half years of operation, the Spartan HPC system has run a total of 20,462,064 jobs at the time of writing (October 3), and has 3,506 accounts across 1,402 projects. The training programme increased from a mere 123 researcher training day's enrolment from 2012-2015, compared to Spartan's 1321 for 2016-2020, plus additional lectures for postgraduate courses. With a recent upgrade the Spartan system now consists of a total of 162 nodes across several partitions, with a peak performance of 1698 Teraflops, which would place it a #283 in the Top500 June 2020 list. The 2,200 physical CPU cores and 1,800 cloud cores were upgraded to 5,100 physical cores. A substantial change in a recent upgrade was changing the file system from CephFS to IBM Spectrum Scale, primarily for reliability purposes [11], increasing storage from 1.3PB to 2.3PB with an additional 550TB of flash. Network upgrades (including physically moving systems) have meant that a significant bottleneck that the system was facing that the general purpose ethernet, relatively inexpensive, good for both low latency and high throughput - has meant significant improvements in data flow between storage and compute nodes, even without significant change to general CPU job profiles. It is said that necessity is the mother of invention. That only applies, of course, if one really puts their nose to the grindstone, dedicates their mind to the task, and does the hard work. In the case of Spartan there was no doubt that we were very inventive and not afraid of breaking conventional wisdoms of what was required for an HPC system. In under five years Spartan has changed from a small experimental system to a tier-one system for Australia. The system has taken advantage of a diverse architecture and novel technologies, bringing together cloud computing as a means of deployment with flexibility and performance through the implementation of RoCE, all making use of core principles of opportunity costs and encouraging throughput over raw potential performance, coupled with an extensive training programme. Spartan provides flexibility, performance, and reliability, along the vectors of compute, data, and training. According to Plutarch, Lycurgus, the legendary founder Sparta's virtues, was asked why the Spartans sacrificed so little to the Gods. His response was appropriately laconic: "So that we may always have something to offer" [12]. The Spartan HPC system serves as an example of what can be done, and what others can do as well. By design, it never needs to be replaced, and it can continue to grow and replaced like the ship of Theseus where eventually all of its components can be replaced and yet it will remain the same system. The Spartan system will always have something to offer. References [1] Keynes, Simon. "Rulers of the English, c.450–1066". In Michael Lapidge; John Blair; Simon Keynes; Donald Scragg (eds.). The Blackwell Encyclopedia of Anglo-Saxon England. Blackwell Publishing, p. 514, 2001 [2] Lafayette, Lev., Tosello, Daniel., Mead, Bernard (2016). Spartan: Performance and Flexibility : An HPC-Cloud Chimera. Presentation to eResearchAustralasia 2016 [3] Solihin, Yan. Fundamentals of Parallel Architecture. CRC Press, 2016 [4] Lafayette, Lev., Sauter, Greg., Vu, Linh, Meade, Bernard (2016). Spartan Performance and Flexibility: An HPC-Cloud Chimera, Presentation to OpenStack Summit, Barcelona, October 27, 2016. Youtube video also available, doi.org/10.4225/49/58ead90dceaaa [5] Telfer, Stig. The Crossroads of Cloud and HPC: OpenStack for Scientific Research, Open Stack, 2016 [6] Lafayette, Lev., Wilcox, Mark., Turnbull, Mitch., Treml, Eric A. Performance Improvements with GPUs for Marine Biodiversity, Presentation to EResearchAustralasia October 2018 [7] Lafayette, Lev. Exploring Issues in Event-Based HPC Cloudbursting. Presentation to HPC Advisory Council, Fremantle, August 2018 [8] Melbourne Data Analytics Platform, Shaping the future of data-intensive research 2019-2020, University of Melbourne https://cpb-ap-se2.wpmucdn.com/blogs.unimelb.edu.au/dist/6/414/files/2020/07/MDAP-Impact-Brochure-%E2%80%94-Pages.pdf [9] Lafayette, Lev., Software Tools Compared To User Education in High Performance Computing. Proceedings of The Higher Education Agenda conference, Gold Coast, May, 2015 [10] Joseph, Earl., et al., Creating Economic Models Showing the Relationship Between Investments in HPC and the Resulting Financial ROI and Innovation — and How It Can Impact a Nation's Competitiveness and Innovation, IDC Special Study, 2013 [11] Liu, J. et al., "Evaluation of HPC Application I/O on Object Storage Systems," IEEE/ACM 3rd International Workshop on Parallel Data Storage & Data Intensive Scalable Computing Systems (PDSW-DISCS), Dallas, TX, USA, 2018, pp. 24-34, 2018 doi: 10.1109/PDSW-DISCS.2018.00005. [12] Plutarch, (trans. John Dryden) Lycurgus, The Internet Classics Archive, Written 75 C.E.