# Cryo-EM transcript _ This is a compiled collection of transcript material, discussion notes, and examples, from a presentation given at the University of New South Wales on Wednesay, October 27, 2021. Slidedeck available at: http://levlafayette.com/files/2021CryoEMUNSW.pdf _ Thanks to Keiran Rowell for inviting me to contribute here. By way of background, I'm a Senior HPC DevOps Engineer at University of Melbourne and have been there since 2015 as one of the original engineers for the Spartan HPC system. Prior to that eight years at the Victorian Partnership for Advanced Computing. Apart from the system administration tasks, I do a lot of researcher education and am a member of HPC Certification Forum steering board, with a few books published about HPC training and related topics. The presentation will start on outlining, at a high-level, the science behind cryo-electron microscopy, before moving onto the data and processing challenges, the beautiful mess that we have at the University of Melbourne and what is being developed, a few examples, and some future challenges and activities. Please feel free to interrupt with questions, suggestions, and especially corrections. Some of this presentation was given at the all-too-brief lightning talk at eResearchAustralasia last week. This one will be somewhat longer and more detailed. ## Background to Cryo-Electron Microscopy Cryo-Electron Microscopy is part of the broader category of structural biology, which is concerned with the molecular structure of biological macromolecules (especially proteins, nucleotides, membranes), how they acquire the structures they have, and how alterations in their structures affect their function. Macromolecules carry out most of the functions of cells, and it is only by coiling into specific three-dimensional shapes that they are able to perform these functions. For several decades, since 1923, X-ray crystallography was the dominant technique for obtaining high-resolution information about macromolecular structure, with enormous contributions by Dorothy Crowfoot Hodgkin, who solved the structures of cholesterol (1937), penicillin (1946) and vitamin B12 (1956), for which she was awarded the Nobel Prize in Chemistry in 1964, and in 1969, she succeeded in solving the structure of insulin. For a long time the use of transmission electron microscopy for biological structure determination methods was limited because of the radiation damage due to high energy electron beams. The idea of using low temperatures would reduce beam-induced radiation damage was raised, but it wasn't until 1981 that Alasdair McDowall and Jacques Dubochet, at the European Molecular Biology Laboratory, reported the first successful implementation of cryo-EM. In 2017, three scientists, Jacques Dubochet, Joachim Frank and Richard Henderson, were awarded the Nobel Prize in Chemistry for developing a technique that would image biomolecules. Single-particle cryo-EM has been mainly used for large protein complexes that cannot be crystallised or have their structures changes through crystalisation, although at a substantially lower resolutions than crystallography. As of October 27, 2020 X-ray crystallography has been used to image 150,494 biological samples and is the dominant technique in biological microscopy, with Cryo-EM far behind at just 6016. However, the introduction and use of direct electron detectors and automation of sample production is making Cryo-EM a potential rival. Prior to direct electron detectors, photographic film and charge-coupled device cameras were used. Film provided relatively high resolution but tedious to use, to put it mildly. CCD cameras were has the convenience of a digital readout, but the resolution achievable with such cameras was relatively poor. With cryo-EM proteins can be seen in all conformations they adopt, unlike in x-ray crystallography in which only a single structure can be determined. Unlike conventional electron microscopy, cryo-EM samples are not dehydrated or stained, which means their structure remains close to the true shape of the hydrated structure in their native environment, and there are no false shapes created by the stain. Freezing samples minimises the radiation damage that samples can suffer as a result of electron irradiation. Frozen samples are also less likely to be damaged by the low pressure/vacuum conditions of the electron microscope. Specimen preparation in cryo-EM is difficult, for optimised ice thickness, and particle distribution. Sometimes proteins adopt preferred orientations that make 3D reconstruction very difficult indeed, sometimes even impossible. Due to a lack of stain and therefore a lack of contrast, images often have a very low signal-to-noise ratio, requiring highly advanced detection hardware and image processing. And, of course, the equipment is very expensive. A cryo-EM experiment begins with a purified protein sample. A protein solution is applied to a sample grid consisting of tiny holes in a film, conventionally made of amorphous carbon, supported by a metal frame, ideally with particles distributed within the grid holes in a variety of orientations. The grid is then plunged into a cryogen such as liquid ethane, flash-freezing it (you don't want ice crystals), trapping the particles and their structure in a film of ice, which also protects the sample from radiation damage when subject to the transmission electron microscope. Whereas crystallization typically locks a protein into its most stable orientation, proteins in cryo-EM samples are free to move around until the moment of flash-freezing. Because cryo-EM is a single-particle technique, such conformational transitions can be captured and studied. Providing the equivalent of multiple pictures of an object from multiple angles while it is moving. Images of multiple copies of the molecule suspended in random orientations in vitrified ice are recorded, based on the interactions of the matter with a beam of electrons. The structure of the sample is determined by combining the different views of the molecule into a 3D model. The phrase 'single-particle' cryo-EM comes from the fact that 2D electron micrographs are taken of individual protein particles on the sample grid. Because very low electron doses must be used in order to avoid damaging radiation-sensitive samples, such 2D projections are too noisy to allow structures to be resolved in atomic detail. The signals can be improved, however, by averaging of a large number of individual particles. Particles are often frozen in random orientations on the sample grid, so the procedure of averaging, alignment, and data merging is not a straightforward process, requiring some fairly sophisticated image processing tools and methods. Once this is done however, an initial 3D map is constructed that is iteratively refined and validated. Finally, the protein sequence is fitted into the 3D map to build a 3D model of the protein. ## Instruments, Project, and Data at UniMelb That's a very high-level description of the physical science, mainly pitched for the HPC team. Now, mainly for the microscopy people, a description of the information science. As you would already know, the computational process is time-consuming and complex, with significant graphics processing requirenents and massive datasets, and a great deal of interaction required. Even a stripped down version of the basic tutorial for RELION (https://github.com/xtreme-d/relion-tutorial-simplified) has 29 different steps, with significant parameters required with each each step and image processed. This includes, for example, include pixel size, electron dose and defocus values. The description of the typical workflow for for the apoferritin protein gives an idea of the sort of data involved; 428 gigabytes just from the original movies (the image, still of the apoferritin protein, in this case from a mouse, is from a different paper), the first stage of the process. Dealing with this combination of processing requirements, data movement, and user-interaction is extremely challenging. Usually, tools like high performance computing can manage at least one (the processing) or two of the three. The opportunity is taken here to follow some of the actions that have been undertaken at the University of Melbourne. The sketch, in true scientific style, was provided by Associate Professor Isabelle Rouiller, Centre Deputy and Node Leader University of Melbourne, ARC Industrial Transformation Training Centre for Cryo-electron Microscopy of Membrane Proteins. There are also four projects with an aggregate of forty-five researchers that make use of the Spartan HPC system for Cryo-EM, especially the GPU partitions for global alignment of movies, 2-D Classification, 3-D Classification, 3-D Refinement, etc The GPU partition is 72 nodes, each with four NVIDIA P100 graphics cards, which can provide a theoretical maximum of around 900 teraflops was funded through ARC LIEF grant LE170100200. The main purpose of the laboratory is to determine 3D structure of macromolecular protein complexes, including ATPases, bacterial transporters and ion channels via Cryo-EM. There are three main instrument - I said two at eReseach but there's one that's owned by Monash University, but the University of Melbourne is the main user. These are a Thermo Fisher Krios, Thermo Fisher Glacios, and Thermo Fisher Aquilos. These are not cheap machines; a Krios can cost more than $6 million USD, and stands at 4m tall (https://cen.acs.org/analytical-chemistry/microscopy/Cheaper-cryo-EM-horizon/98/web/2020/11). In terms of data collection, there are roughly 140 instruments at the University of Melbourne, according to the recent project of the Petascale Campus Initiative. The three Cryo-EM instruments account for over 50% of the total instrument data, with 4.8 terabytes a day, c2.32 petabytes a week. This data is transferred to a MediaFlux storage, which it is typically processed by a dedicated high-performance (and high-cost) GPU Linux server. However, graphical throughput is an issue when connecting from outside the campus. During the recent 'lockdown' period in Melbourne, residential connections to this server has been an issue. This is despite the fact that it the connections are made with a cache/forward system, NoMachine, rather than X-Windows forwarding. In particular there is significant interactive visualisation required through ChimeraX. Whilst this is typically carried out through the specialised Linux server, the Spartan HPC system is also used as well. Mediaflux is fairly well integrated with the LMod enviroment modules system, which provides the paths and description: ``` $ module avail unimelb-mf-clients ------------------------------------------------------------------------ Core Modules ------------------------------------------------------------------------ unimelb-mf-clients/current $ module load unimelb-mf-clients $ module display unimelb-mf-clients/current ---------------------------------------------------------------------------------------------------------------------------------------------------------- /usr/local/easybuild-2019/easybuild/modules/all/core/unimelb-mf-clients/current.lua: ---------------------------------------------------------------------------------------------------------------------------------------------------------- help([[ Description =========== unimelb-mf-clients is a collection of command-line tools to transfer data to/from Mediaflux, the data management platform operated by Research Computing Services More information ================ - Homepage: https://wiki-rcs.unimelb.edu.au/display/RCS/Unimelb+Command-Line+Clients ]]) whatis("Description: unimelb-mf-clients is a collection of command-line tools to transfer data to/from Mediaflux, the data management platform operated by Research Computing Services") whatis("Homepage: https://wiki-rcs.unimelb.edu.au/display/RCS/Unimelb+Command-Line+Clients") conflict("unimelb-mf-clients") prepend_path("PATH","/usr/local/easybuild/software/unimelb-mf-clients/current/bin") setenv("MFLUX_ATERM","/data/gpfs/admin/unimelb-mf-clients/current/lib/aterm.jar") ``` Data is transferred from Mediaflux to Spartan, generally to either a project directory or scratch space, as these have superior throughput and size (500 GB) compared to the home directories (50GB by default). The Mediaflux clients module is loaded which provides multi-threaded (up to 4 threads) command-line Mediaflux access. In addition to the MediaFlux clients module there is also the MediaFlux data mover and explorer modules. The data mover modul can be used to transfer data between Mediaflux and a file system using pre-created shareable links (for upload and download). The MediaFlux explorer module is a GUI client to manage data which can be accessed through a FastX sesssion. ``` $ module av mediaflux ------------------------------------------------------------------------ Core Modules ------------------------------------------------------------------------ mediaflux-data-mover/current mediaflux-explorer/current ``` The relevant software we have built on Spartan includes: EMAN2: Greyscale scientific image processing suite with primarily for processing data from transmission electron microscopes, RELION (for REgularised LIkelihood OptimisatioN) employs an empirical Bayesian approach to refinement of 3D reconstructions or 2D class averages in electron cryo-microscopy, Scipion is an image processing framework for obtaining 3D models of macromolecular complexes using Electron Microscopy (3DEM). It integrates several software packages (such as Xmipp, Relion, Spider, Eman2, Bsoft, Frealign, Ctfind, etc) and presents a unified interface, and cisTEM: Process cryo-EM images of macromolecular complexes and obtain high-resolution 3D reconstructions. This sortware was build using the EasyBuild framework, which is highly recommended in most cases and integrates well with the LMod environment modules sytem. Such software will vary according to the version and the compiler toolchain. In our earlier implementation this would be part of the build file and in the software module title, which could end up being uncomfortably long. In a more recent implementation, we force the researcher to load a compiler toolchain first and then select the version. Despite our best efforts, too many researchers would ignore toolchain conflicts and simply load whatever software modules they wanted, resulting in whatever was load last as being the toolchain; this often leads to results which are difficult to reproduce. For example: ``` $ source /usr/local/module/spartan_old.sh $ module av RELION -------------------------------------------------------------- /usr/local/easybuild/modules/all -------------------------------------------------------------- Relion/2.0.3-intel-2016.u3 Relion/3.0-intel-2017.u2-GCC-6.2.0-CUDA9 Relion/3-1-spartan_intel-2017.u2 (D) Relion/2.0.3-intel-2017.u2-GCC-6.2.0-CUDA9 Relion/3-0-intel-2017.u2 Relion/3-beta-intel-2016.u3 Relion/3-0-spartan_intel-2017.u2 $ less /usr/local/easybuild/ebfiles_repo/Relion/Relion-3.0-intel-2017.u2-GCC-6.2.0-CUDA9.eb ... $ source /usr/local/module/spartan_new.sh $ module av relion ------------------------------------------------- Toolchain: foss/2019b Compiler: gcc 8.3.0 OpenMPI: 3.1.4 -------------------------------------------------- relion/3.0.4 relion/3.1.0 (D) ----------------------------------------- Toolchain: fosscuda/2019b Compiler: gcc-cuda 8.3.0-10.1.243 OpenMPI: 3.1.4 ----------------------------------------- relion/3.1.0-test relion/3.1.0 ... $ module load relion/3.1.0 Lmod has detected the following error: Can't load relion/3.1.0 because it has more than one parent hierarchy, making this load ambiguous, or loaded toolchain doesn't include relion/3.1.0 ... $ module load foss/2019b $ module load relion/3.1.0 ``` Some examples of job scripts run through the HPC system in different environments (e.g., Slurm, Jupyter notebooks) include the following ``` $ less /data/projects/punim0421/MohsenProjects/projects/importMovie/Runs/000470_XmippProtMovieGain/logs/470.job $ less /data/projects/punim0979/sehee_kim/script_ctla4_mono.slurm ``` Most recently started using SBGrid execution environment, as software toolchains are often highly customised. The SBGrid Consortium operates from Harvard University and they they provide a suite of some 400+ pre-compiled structural biology applications with, following registration, a simple-command line installation manager (`sbgrid-cli`) monthly updates and testing, individual applications can be self-managed, with a regular rsync. Specific applications are determined by a database for customized download lists. The environment is simply invoked by a simple source command and can be easily integrated into HPC job submission script templates. Sufficiently good that it has replaced use of our custom-built versions of Scipion and Relion. The environment is simply invoked by a simple source command and can be easil integrated into HPC job submission script templates. It is sufficiently good that it has largely replaced use of our custom-built versions of Scipion and Relion. For a tool like Relion, or others that require graphics-intensive interactive work, we recommend invoking it through a FastX environment, on the small "interactive" partition. Jobs can be launched following the FastX webpage (https://spartan-fastx.hpc.unimelb.edu.au/) which then provides an XFCE Linux desktop environment from which a virtual terminal can be launched with the appropriate commands entered. A simple test simply using X-Windows forwarding can be illustrated as follows: ``` $ sinteractive --mem=32G --x11 --time=01:00:00 -p interactive $ cd /data/projects/punim0584/sbgRelionTest/Import/job001 $ cat note.txt ++++ Executing new job on Thu Jul 1 16:08:53 2021 ++++ with the following command(s): relion_import --do_movies --optics_group_name "opticsGroup1" --angpix 1.3 --kV 300 --Cs 2.7 --Q0 0.1 --beamtilt_x 0 --beamtilt_y 0 --i "*.tif" --odir Import/job001/ --ofile movies.star --pipeline_control Import/job001/ ++++ ``` The X-Windows forwarding issue is significant when a user's graphics forwarding requirements is distal from where the computation occurs There are several solutions to this problem. This includes simpler graphics, a simpler desktop, or make use of cache/forward alternatives such as X2go and NoMachine NX Technology (which was used on UniMelb's previous system, Edward). This can also involve VirtualGL that redirects the 3D-rendering commands from Linux OpenGL applications to 3D accelerator hardware. An alternative range of remote desktop software use Remote Frame Buffer (RFB) protocol used in Virtual Network Computing (VNC), such as TigerVNC, TightVNC, and Cendio's ThinLinc. The University of Melbourne did use ScienTific Remote Desktop Launcher (Strudel) for a period, a TurboVNC-based system developed originally at VPAC, then Monash University, that has deployed at several Australian research institutions, including NCI, Pawsey, Monash University. More recently, UniMelb has deployed FastX from Starnet (founded in 1989) which has excellent performance. Whilst the technology is proprietary Starnet does advertise that their performance is due to well-known approaches such as remote server rendering, improved compression algorithms, monitoring of local graphic application changes, automatic or manual adjustment of frame-rate depending on network speed, and frame windowing for interactive applications. The cryo-electron microscopy laboratory would really prefer to do their workload on Spartan, but in most cases they require GPU access. But we (currently) don't have FastX available for the GPU partition, and it is faster than the alternatives see: (https://www.starnet.com/fastx/performance). We do have the GPU partition available with Spartan's Open OnDemand and Jupyter Notebook are used by the appropriate projects. Even when it is implemented physical distance will be an issue; FastX is much quicker than X-Windows forwarding, but it is not the same as having a local system. A general principle of high-performance computing has been "compute remotely, visualise locally", itself coming from an even stronger principle of keeping datasets and processing as close as possible. Admiral Grace Hopper's presentation "Mind your nanoseoconds" is a wonderful and pithy description of the issue (`https://www.youtube.com/watch?v=9eyFDBPk4Yw`) ``` $ source /programs/sbgrid.shrc ... $ cd /programs/x86_64-linux/ $ ls $ cd /data/projects/hpcadmin/lev/relion $ /programs/x86_64-linux/relion/3.1.2_cu9.2/bin.capsules/relion ``` ## Future Plans and Acknowledgements In the future we are looking at those structural biology tools that assume the presence of a website (e.g., cyroSPARC). The main uses will be (a) aligment of CryoEM microscope, where they need to realtime stream the data to CyroSPARC Live and (b) Once the microscope is aligned, the data is taken and streamed to Mediaflux. The ownership is transferred to the specific research group the microscope operators. As this is something that is typical in an HPC environment it is probable that we will use a virtual machine. The main issue is that we don't have dedicated nodes on the HPC for such purposes, so we're building it with Cloud VMs with the challenge of data movement from microscope to VM and ownership. As cryoSparc runs as a specific user, all data needs to be accessible to that user. Because the services run as X user, you would need to spin up N instances of the Cryosparc GUI for the N users who want to use it As mentioned, we are also very keen to have an visual interactive partition for GPUs using FastX or even some other X-Windows environment. However, using the systems that we currently have would not be allowed under the conditions of the LIEF grant, so that will have to wait until we have new GPU-enabled systems online that are independent of such conditions. Finally, there is a real need for a proper example tutorial with HPC integration. We do have a good collection of tutorials for job submission (see `/usr/local/common`) for other applications, our tutorials for the Cryo-EM tools is woefully underdeveloped. I would like to thank the following people for their assistance in this presentation. Assoc. Prof. Isabelle Rouiller, Centre Deputy and Node Leader University of Melbourne, ARC Industrial Transformation Training Centre for Cryo-electron Microscopy of Membrane Proteins (CceMMP) Sean Crosby, Senior DevOpsHPC Engineer and HPC Team Lead, Research Computing Services, University of Melbourne Andy Tseng, Head of Data Solutions, Research Computing Services, University of Melbourne