Saturday, November 3, 2012

General Relativistic Simulations of Three-Dimensional Core-Collapse Supernovae (arXiv:1210.6674) -- Meet the Team!

A bit more than a week ago, we uploaded our first full 3D core-collapse supernova paper to the arXiv (arXiv:1210.6674). Making the simulation code work, getting the simulations done, analyzing and understanding them, making the (I think, breathtaking) graphics, and writing everything up was a huge team effort. Ultimately, the paper will be referred to as Ott et al. (2012), which, I find, does not give enough credit to all the other team members whose contributions were crucial.  In this blog post, I want to introduce and give due credit to my team.

The author list: Christian Ott, Ernazar Abdikamalov, Philipp Mösta, Roland Haas, Steve Drasco, Evan O'Connor, Christian Reisswig, Casey Meakin, and Erik Schnetter.
(Important aside: You have probably noticed that there is not a single woman on the author list. The extreme underrepresentation of women in theoretical and computational astrophysics is a big problem that is hurting our field.)

My main contributions were: Defining the scope of the project, doing the actual simulations (which were carried out on supercomputers at NERSC, NICS, and TACC), overseeing the analysis and visualization, and writing the paper.

Ernazar Abdikamalov, Postdoctoral Scholar
Ernazar Abdikamalov, TAPIR, Caltech
Ernazar Abdikamalov --  Ernazar was the analysis czar for this paper and he also wrote the first draft of the section in the paper discussing explosion criteria. He wrote (together with Roland Haas) the analysis package and carried out the time consuming postprocessing analysis of our 3D simulation data. For each of our simulations, we kept about 5-8 Terabytes of output (in HDF5 format) and digging out the information needed for the paper meant reading in time slices of 3D output and running the analysis code on them. One full postprocessing run took about a day, but Ernazar spent weeks re-doing the postprocessing, since we kept on adding things we wanted to look at as the paper progressed.

Some background on Ernazar: I first met Ernazar (who originally comes from Uzbekistan) when he was a PhD student working with Luciano Rezzolla at SISSA at Trieste. I ended up collaborating with Ernazar on the final paper (on general-relativistic simulations of accretion induced collapse) towards his PhD thesis in 2009. He then went to work as a postdoc with my friends in the numerical relativity group at LSU's Center for Computation and Technology. Knowing the outstanding work he does, I brought him to Caltech first as a long-term visitor, then hired him as a postdoc, once I had the money to do so. His current primary research interest and specialty is radiation transport and we have a few papers together on this and other subjects.

Philipp Mösta, TAPIR, Caltech
Philipp Mösta -- Philipp was the plot czar for this paper. He worked very closely with me and Ernazar in making sense of the simulation results and presenting them in the graphs and 2D color maps shown in the paper. These plots were made with the open-source and Python-based matplotlib package and after the hundreds of revisions he made to the plots, Philipp is a true expert and knows how to make excellent plots (which is a skill that must be acquired by hard work). Philipp also participated closely in the discussions led by Ernazar and myself of what our results actually mean and how to present them in the paper.

Some background on Philipp: Philipp is a recent PhD (early 2012) coming out of Luciano Rezzolla's group at the Albert Einstein Institute (the Max Planck Institute for Gravitational Physics in Potsdam, Germany). I have known him since 2006, when he was at the AEI working on a numerical relativity code for his master's thesis. His PhD research was on electromagnetic fields and their interaction with merging binary black hole systems. He arrived at Caltech in November 2011 and is a postdoc in my group. His main interest right now are magnetohydrodynamic processes in core-collapse supernovae, but he continues to work on more fundamental numerical relativity problems and also helps with double neutron star merger simulations using the SpEC code of the Simulating eXtreme Spacetimes collaboration.

Roland Haas, TAPIR, Caltech
Roland Haas -- Roland was the key instrumentalist for this paper. Simulation codes are research instruments, in the same sense as instruments in experimental research. Instrumentalists in experimental physics (for example the instrumentalists who make LIGO work) always get credit for their work and it is time to extend this to computational research. Let's face it: Without people like Roland who know the ins and outs of the complex 3D codes that are now being used by us and other groups to model cosmic explosions (and many other things) our work would just not be possible. Our core collapse simulation package Zelmani is based on the open-source Einstein Toolkit, which in turn uses the Cactus Computational Toolkit (as its base infrastructure) and the Carpet adaptive mesh refinement driver. Roland is one of the key developers of all these codes (i.e. scientific instruments) and knows them by heart. His contribution was crucial for getting the code to work and ironing out infrastructure-related problems, in particular with the new multi-block grid that our simulations used for the first time. Roland also made important contributions to Ernazar's postprocessing work. They are best summarized by the little running joke we have in our group: "How does Ernazar program Cactus?" Answer: "He walks into Roland's office."

Some background on Roland: Roland received his PhD from the University of Guelph, where he worked with Eric Poisson on self-force calculations in general relativity. He then worked in Pablo Laguna's group at Georgia Tech, where he became involved in the Einstein Toolkit. In 2010, he won a Canadian NSERC postdoctoral fellowship, which he took to Caltech in the fall of 2011. His main current research is focused on the double neutron star merger problem, which he is tackling with the SpEC code. He has also been working closely with Christian Reisswig on the new multi-block extension to Zelmani.

Steve Drasco, Grinnell College
Steve Drasco -- Steve did all the 3D rendering and generated 2D entropy colormaps shown in the paper. He is also the guy responsible for the movie of one of our simulations that's available on the Simulating eXtreme Spacetimes YouTube channel. This was extremely time consuming, but also extremely important work, which he carried out with the open-source VisIt visualization package. Steve not only rendered the final simulation output, but was frequently called to help to diagnose potential (and real) problems as the simulations were progressing. Thanks to him, we identified a problem appearing when the supernova shock gets too close to AMR boundaries. Had we not looked at his plots, we would not have noticed it and would have wasted a huge amount of supercomputer time.

Some background on Steve: Steve is a relativist by training and received his PhD from Cornell, working with Eanna Flanagan. His main research focus has been on extreme and intermediate mass ratio binary black hole inspirals. He came to Caltech/JPL as a postdoc in 2005, moved to the Albert Einstein Institute in 2008, was a lecturer at CalPoly San Luis Obsipo from 2010 until this June. He is now an Assistant Professor of Physics at Grinell College and has a visitor appointment in my group in TAPIR. Steve has always had a great talent for visualizing his work. Some of his stuff has recently been featured in Physics Today. Also check out his cool movies of these inspirals. Steve loves fast cars and owns two Porsches.

Dr. Evan O'Connor (right) and Christian Ott (left).
Evan O'Connor -- Evan was the microphysics expert for our paper. He put together the equation of state table used in our simulations. Evan also developed the leakage scheme originally used in his code (GR1D) and helped porting it to 3D and integrating it in Zelmani. In order to set the stage for our 3D simulations, he ran a suite of spherically-symmetric simulations with GR1D for us to build some intuition for the overall collapse and postbounce dynamics of the 27-solar-mass progenitor that we used in our simulations.

Some background on Evan: Evan is the first PhD that I produced and I am as proud as a "parent" could possibly be! The picture on the right shows Evan and me on the patio of Caltech's Cahill Center for Astronomy & Astrophysics right after his defense. The hat that we made for him says "Licensed Supernova Theorist". Evan graduated this June and, since September 1, he is a postdoctoral fellow at the Canadian Institute for Theoretical Astrophysics at the University of Toronto. His main current interests are neutrino transport, neutrino microphysics, and the nuclear equation of state. Check out his publication list. It's already quite impressive!

Christian Reisswig, TAPIR, Caltech
Christian Reisswig (The Reisswig) -- Christian's role in this paper was that of an instrumentalist. He is one of the co-developers of the Llama Code, an extension to Cactus and Carpet that latches multiple grid blocks with logically Cartesian, physically curvilinear coordinates onto Carpet's AMR grids. This is a great technical advantage over previous, purely Cartesian 3D approaches, because it allows us to push the outer boundary of the computational domain out to very large radial distances while keeping the angular resolution fixed and thus saving a tremendous amount of memory and supercomputer time. Christian is also an expert of our Carpet AMR driver and has been working very closely with Roland Haas and Erik Schnetter on improving various aspects of it. He is also our lead optimizer: We could not have run our simulations without his improvements to the parallel performance of our code

Some background on Christian: I have known Christian since ~2005, when he arrived as a master's student at the Albert Einstein Institute (where I was a PhD student at the time). Already back then, he played around with an early multi-block infrastructure. In his PhD research (which he concluded in 2010), he worked on binary black hole inspiral and merger simulations and became one of the leading experts in extracting gravitational waves from such simulations. He arrived as my first postdoc at Caltech in early 2010 and was awarded an Einstein Fellowship in 2012. His main research interests right now are black hole formation, his magical next-generation Simsalabim high-performance computing infrastructure (ask him about it after he has had a few drinks!), and rapidly rotating core collapse.

Casey Meakin, LANL
Casey Meakin -- Casey's contributed to our paper by working with Ernazar and me on the analysis of neutrino-driven convection in our models. He also worked out what the typical magnitude of perturbations (local deviations/fluctuations from the spherical background) one would expect to be presented in the core of the 27-solar-mass presupernova star whose collapse we simulated. This was crucial, because it gave us a quantitative handle not only on the size of these physical perturbations (which will seed convection), but also on the time they will reach the stalled supernova shock. This helped us to argue that the relatively large numerical perturbations that our simulations experience (from AMR boundaries and the Cartesian grid) are comparable to what one would expect from convective burning in silicon core and shell burning.

Some background on Casey: I have known Casey since 2001 and he is one of my few close long-time friends. We were both graduate students at Steward Observatory at the University of Arizona (well, I was sort of an "exchange student") at that time working with Adam Burrows. Casey subsequently worked with Dave Arnett on multi-dimensional stellar evolution and graduated in 2006. He was a postdoc at the Flash Center at the University of Chicago, then went back to Arizona for a while and is now a Scientist in the Theoretical Division of Los Alamos National Laboratory and adjunct faculty at Steward Observatory.

Erik Schnetter, Perimeter Institute
Erik Schnetter -- It is hard to adequately summarize Erik's contribution to this paper; his work on Cactus and, in particular, on the Carpet AMR driver is enabling everything we do with Zelmani. Perhaps, I can describe him as the senior instrumentalist in our team. His instant messenger screen name is eschnett247 -- and he has been a tremendous near-24/7 resource for infrastructure help and guidance on how to get our simulations to perform well on various supercomputers. Earlier this year, I think it was in May, when we were gearing up to start our simulations, Erik dedicated an entire weekend to helping us in improving parallel scaling of the production-level simulation setup. We reserved half of our local Caltech supercomputer (which is called Zwicky, in honor of late Caltech Astrophysicist and supernova hunter Fritz Zwicky), my crew at Caltech got together, we established a video conference with Erik and off we went on hacking, breaking, fixing, and improving Zelmani. Erik also did a tremendous amount of extremely difficult code design and development that enabled our simulations: He designed and implemented cell-centered AMR in Carpet and a mechanism called "refluxing", which allows for exact conservation of mass, momentum, and energy fluxes at AMR boundaries.

Some background on Erik: I have known Erik since 2003 and he is another very close, long-time friend. I met him when he was a postdoc at the Albert Einstein Institute and I was a graduate student there. Erik (together with Ian Hawke) basically trained me in numerical relativity and computational science. In 2005, he moved as a Research Assistant Professor and Staff Scientist to the Center for Computation and Technology at LSU and in 2010 he became Research Technologies Group lead at the Perimeter Institute in Waterloo, Ontario, Canada. He is also adjunct faculty at the University of Guelph.

No comments:

Post a Comment