188宝金博页面版

Petascale Direct Numerical Simulation of TurbulentChannel Flow on up to 786K Cores∗Myoungkyu LeeDepartment of MechanicalEngineeringUniversity of Texas at AustinAustin, Texas 78735mk@ices.utexas.eduNicholas MalayaInstitute of ComputationalEngineering and SciencesUniversity of Texas at AustinAustin, Texas 78735nick@ices.utexas.eduRobert D. MoserInstitute of ComputationalEngineering and Sciencesand Department ofMechanical EngineeringUniversity of Texas at AustinAustin, Texas 78735rmoser@ices.utexas.eduABSTRACTWe present results of performance optimization for directnumerical simulation (DNS) of wall bounded turbulent f l ow(channel f l ow). DNS is a technique in which the f l uid f l owequations are solved without subgrid modeling. Of particu-lar interest are high Reynolds number (Re) turbulent f l owsover walls, because of their importance in technological ap-plications. Simulating high Re turbulence is a challengingcomputational problem, due to the high spatial and tempo-ral resolution requirements.An optimized code was developed using spectral methods,the method of choice for turbulent f l ows. Optimization wasperformed to address three major issues: ef f i ciency of bandedmatrix linear algebra, cache reuse and memory access, andcommunication for the global data transposes.Results show that performance is highly dependent on char-acteristics of the communication network, rather than single-core performance. In our tests, it exhibits approximately80% strong scaling parallel ef f i ciency at 786K cores relativeto performance on 65K cores.∗ Permission to make digital or hard copies of all or partof this work for personal or classroom use is granted with-out fee provided that copies are not made or distributed forprof i t or commercial advantage and that copies bear this no-tice and the full citation on the f i rst page. Copyrights forcomponents of this work owned by others than ACM mustbe honored. Abstracting with credit is permitted. To copyotherwise, or republish, to post on servers or to redistributeto lists, requires prior specif i c permission and/or a fee. Re-quest permissions from Permissions@acm.org.SC13, November 17-21 2013, Denver, CO, USACopyright 2013 ACM 978-1-4503-2378-9/13/11...$15.00.http://dx.doi.org/10.1145/2503210.2503298Categories and Subject Descriptors[Algorithms]: Numerical methods, linear and non-linearsystems; [Applications]: Computational f l uid dynamicsGeneral TermsTurbulence, MPI alltoall, Parallel FFT, Petascale, Data trans-pose1. INTRODUCTIONTurbulence is a multi-scale f l uid f l ow phenomenon character-ized by large unpredictable f l uctuations that occur across awide range of length and time scales. Described by RichardFeynman as, “the most important unsolved problem of clas-sical physics.”, the turbulence problem remains unsolved de-spite over one hundred years of scientif i c research. Never-theless, approximately 20% of global energy consumptionis expended on transportation[26], in which vehicles movethrough the air or water, or f l uids are transported throughpipes and ducts, and this energy is dissipated primarily inturbulence. Unfortunately, there is currently no theory ormodel of wall-bounded turbulence of suf f i cient veracity tosupport the development of new ef f i cient designs of turbu-lent f l uid systems.For Direct Numerical Simulations (DNS) of turbulence, theequations of f l uid motion (the Navier-Stokes equations) aresolved, without further modeling, with suf f i cient temporaland spatial resolution to represent all the scales of turbu-lence. Such simulations provide exquisitely detailed andhighly reliable data, which have driven a number of dis-coveries regarding the nature of turbulence [10, 8, 28, 3, 13].Indeed, for some quantities that are dif f i cult or impossibleto measure, DNS is the most reliable or only source of data.The turbulent f l ow of f l uids past walls at high speed (highReynolds number) is particularly important in the trans-portation examples mentioned above, as well as many otherapplications. The Reynolds number measures the relativeimportance of inertia relative to viscosity, so that the higherthe Re, the weaker the stabilizing ef f ect of viscosity. How-ever, the use of DNS to study f l ows has been hindered bythe extreme computational expense of high Reynolds num-ber turbulence simulation. The ratio of the size of the largestto the size of the smallest turbulent eddies in a f l ow increaseswith the Reynolds number. In wall-bounded turbulence like