Computer Applications in Science & Engineering Department

From AnnualReport

Jump to: navigation, search
José Maria Cela, Director of the CASE Department

The aim of the Computer Applications in Science and Engineering (CASE) Department is to develop new computational strategies to simulate complex problems specifically adapted to run efficiently on modern supercomputers. Collaborative projects with industry and scientific groups are the main motivation underlying all development carried out in CASE.


The applications developed by the CASE department are truly multidisciplinary, requiring a deep level of expertise in many fields. In order to successfully develop these applications, the skills of the CASE team in numerical methods and parallel programming must be complemented by experts in the appropriate areas. The Department therefore develops collaborations with other scientific groups in all areas of science and technology. Examples of Spanish institutions with strong research links with CASE include CIEMAT, CSIC, IAC, ICFO, IMDEA and different universities. CASE also collaborates with institutions abroad like Imperial College, Oxford University, STFC in the UK, EDF and Ecole Centrale de Paris in France, and George Mason University and Jackson State University in the USA. This is complemented with strong links with industrial partners in need of advanced simulations of complex technology problems, such as REPSOL or Iberdrola.

The main research field of CASE is High Performance Computational Mechanics, which requires a deep background in Computer Science, Physics and Numerical Methods. Major research areas are Computational Fluid Dynamics and Solid Mechanics, Ab-initio DFT and TD-DFT molecular dynamics, Seismic Imaging and Parallel Programming. Major application areas are Aerospace, Fusion Physics (plasma core and edge transport, plasma inestabilities), Biomechanics (Cardiovascular and Respiratory systems), Geophysics and Atmospheric flows. CASE also has a group working in large scale social simulations and smart cities. Finally, CASE has a group devoted to visualisation tools and techniques, which are critical to extract full benefit from numerical simulations.

To achieve its objectives, the CASE team develops and co-develops five main high performance codes, which are used in national/international projects and are at the core of CASE's collaborations and contracts with companies:

  • Alya: HPCM system. Fluid mechanics, Solid mechanics, Electric propagation, Combustion, etc.
  • FAll3D: Volcanos ash transport. Used in production in South American Volcanic Ash Advisory Centres (VAAC)
  • BSIT(Barcelona Subsurface Imaging Tools): Acoustic/Elastic/EM waves, Forward Modelling, RTM, FWI. Promoted by Repsol.
  • SIESTA: Ab-initio molecular dynamics. CASE is a co-developer of this code.
  • Pandora: An HPC Agent-Based Modelling framework for social simulation.

Organisational Structure

The CASE Department is led by José María Cela. The research lines fall naturally in six main groups: Physical & Numerical Modeling (PNM), High Performance Computational Mechanics (HPCM), Environmental Simulations (ES), Geosciences Applications (GA), HPC Software Engineering (HPCSE), and Data Pre & Post Processing (DPPP). Each Group consists of around 10 people, comprising several senior scientists, post and pre-doctoral students and visiting scientists. PNM, HPCSE and DPPP research lines are horizontal, in the sense that they develop core components and tools used by other groups. HPCM, ES and GA lines are vertical, in the sense that they develop applications tailor-made to meet specific project needs. Due to the multidisciplinary character of CASE research activities several groups are often involved in each project.

Key Projects

In 2014, the CASE Department carried out work under the scope of the following projects:

  • EU-funded projects:

CASE was actively involved in PRACE3IP and PRACE 4IP projects, mainly concerning the optimisation on massively parallel supercomputers of Alya code, which is part of the European Unified Benchmark Suite. Also, important contributions were made to applications work packages of DEEP and DEEP-ER.

  • Enterprise-funded projects:

Iberdrola and Repsol are CASE's main industrial collaboration partners in the energy sector. Also, CASE actively participates in the Exascale Lab funded by Intel on topics related to Geophysics.

  • Nationally funded projects:

CASE was involved in the S4E (Supercomputing for Energy) project, which aims to develop high performance simulation codes in the area of energy, namely wind energy and oil and gas.

  • The CASE department also develops international/national collaboration projects in the area of biomechanics:

CASE has established strong national and international collaborations, all of them fostered by the Severo Ochoa Programme. In summary: A CASE post doctoral researcher spent nine months as resident researcher at the Mount Sinai Hospital in New York, starting a research line on carotids simulations; an MOU was signed with the National Centre for Cardiology Research (CNIC) to work on ventricular arrhythmias, with a BSC-CNS PhD student co-supervised by CNIC; CASE is performing full cardiovascular simulations from the heart up to the cerebral arteries by coupling Alya with ADAN, the arterial network created at the LNCC in Brazil; a collaboration with Imperial College and St Mary's hospital (UK) conducted respiration and sniff simulations with an unprecedented degree of accuracy; a collaboration with Jackson State University coupled real patient large airways to generic models for small airways, in order to carry out simulation of the almost complete respiratory system.

  • Other collaborations:

The department is collaborating in various topics with the following institutions:

  • CINES, France: asynchronous and dynamically load balanced code coupling.
  • Technical University of München, Germany: multi-code coupling strategies.
  • STFC, UK: multi-code coupling strategies.
  • EDF, France: multi-code coupling strategies.
  • NCSA, USA: performance studies of Alya code on Blue Waters supercomputer and scalability of direct solvers.

Scientific Output

For additional information, please see the Detailed Report of Research Activities 2014 for the CASE Department.

Except for work that is private and confidential and cannot be published, research results of the CASE Department were presented in congresses and conference lectures as well as a number of scientific publications, including:

  • Development and optimization of a solid mechanics module in Alya.
  • Implementation of coupling strategies in a distributed memory environment (FSI).
  • Coupling of Alya and Code_Saturne, the CFD code developed at EDF, France.
  • Development of an implicit VMS compressible solver and preconditioning techniques.
  • 3D Seismic inversion for large data sets using FWI from BSIT.
  • Development of electromagnetic modelling (CSEM) on BSIT.
  • BSIT kernel optimization for both Intel Sandy Bridge processors and Intel Xeon Phi co-processors.

Research Groups

Physical and Numerical Modelling (PNM)

Computational Mechanics

The PNM Group researches basic themes, such as numerical modelling of physical phenomena, stabilisation techniques, algorithms and solution strategies, parallelisation strategies, coupled problems with domain decomposition methods, optimisation algorithms and error estimation techniques. In addition, PNM researchers investigate pre-process, post-process, data management and visualisation topics. The research lines within PNM cover the full range of techniques required to simulate a physical problem, usually governed by partial or ordinary differential equations. The main areas of investigation are:

  • Mathematical modelling of a given physical process.
  • Numerical modelling of the mathematical equations.
  • Numerical algorithms to solve the discrete equations efficiently, or to couple a set of algorithms to solve complex physical problems.
  • Efficient implementation in a computational mechanics code.
Chimera method applied to the structure dynamics of a neurone using the HERMESH method
  • Code performance analysis and optimisation.
  • Multiphysics coupling. Asynchronism.
  • RANS turbulence models. Models specially designed for wind farm applications (Iberdrola).
  • A large-strain solid mechanics simulation for anisotropic cardiac tissue.
  • Lagrangian particle tracking. Drag and Saffman forces, Brownian motion.
  • ALE method for mesh motion.
  • Chemical transport through interfaces.
  • Rigid body-fluid coupling.
  • Variational multiscale methods for compressible flows.
  • Domain composition methods: development of the HERMESH method applied to Chimera and mesh gluing.
C2CA project: ignition sequence of the gas phase in a rotary kiln using large-eddy simulation (LES)

Due to the installation of new large scale supercomputers in Europe during recent years, the Group dedicated a lot of resource to upgrade the Alya high performance computational mechanics (HPCM) code, including:

  • Solvers.
  • Sparse direct solver.
  • Implementation of the restricted Additive Schwarz preconditioner.
  • A parallel version of SIESTA code with better load balancing and sparse iterative eigensolvers.
  • Speed up tests were carried out on the main European supercomputers on up to 22528 CPUs.

Computational Social Sciences

Since 2009, the Group has worked on the design of applications specially designed for use in social sciences and policy analysis areas. The Group is developing a new simulator capable of executing Agent-Based Models of human societies in an HPC environment, in order to explore:

  • Emergence of behavioural patterns in human societies, understood as complex systems.
  • Interaction between societies and their relationship with environment and landscape.
  • Impact of change in human groups and population dynamics (both ancient and present).
  • Design of artificial societies as models to understand human behaviour.
  • Methodological and theoretical foundations of social simulation.

These topics are analysed from a multidisciplinary approach through collaborations with research groups belonging to different disciplines with diverse perspectives of social interaction (i.e. Archaeology, Demography, Economy, Heritage, History and Sociology).

High Performance Computational Mechanics (HPCM)

The HPCM Group conducts application research and development in different science and technology domains where simulations are needed: aerospace, bio-mechanics, solid state physics, high energy physics, geophysics, environment, meteorology, etc..

Activities are driven by direct interaction with users and industry. Usually the core problem requires modelling of physical processes which then must be solved by intensive numerical calculation. The principal application fields that have been developed to date are:

  • Alya applications:
    • Biomechanics: hemodynamics, respiratory system air flow, cardiac simulations.
    • Building, energy and environment: mesoscale, urban environments, wind farms, plastics recycling.
    • Vehicle dynamics: cars, racing yachts, high speed trains.
    • Simulation of chemical reactions in biodiesels inside batch reactors, coupling with mixing blades, transfer of chemical species through the interfaces of immiscible liquids.
  • Fall3D applications:
    • Atmosphere science: Volcanic ash transport.
  • SIESTA Applications:
    • Ab-inito DFT and TDDFT molecular dynamic simulations.
  • Other applications:
    • Plasma physics.

HPC Software Engineering (HPCSE)

The HPC Software Engineering team is responsible for the good performance of the different codes developed by CASE. This group designs the applications software architecture and performs fine tuning of these codes on different hardware architectures. The applications are designed in such a way that all possible levels of parallelism are exploited and aspects such as fault tolerance are taken into account. For the software tuning this group collaborates with hardware manufacturers to obtain roofline analysis of the codes and other performance metrics. Aspects like documentation, testing, etc, are also responsibility of this group.

Environmental Simulations (ES)

Atmospheric Transport

Modelling of atmospheric transport, with particular emphasis on volcanic ash. Research lines include:

  • Volcanic Ash Transport and Dispersal Models (VATDMs), including model validation, ensemble forecast and operational implementation.
  • Development of theoretical models for ash aggregation, dynamics of volcanic plumes, gravity currents, and resuspension of ash by wind.
  • Assessment of hazard and impact of volcanic ash fallout on local communities and of volcanic ash clouds on civil aviation.
  • Study the feedback effects of large-magnitude eruptions on regional meteorology. This is done in collaboration with the Earth Science Department and using FALL3D-NMMB/BSC-CTM, an on-line multi-scale meteorological model coupling the dispersion and sedimentation functionally of FALL3D with the powerful NMMB/BSC-CTM weather forecasts from global to mesoscale domains.
  • Code optimisation. Implementation of transport models in multi-purpose frameworks and porting of parallel software to different architectures.
Comparison between forecasted ash cloud column mass (ton km-2) and split windows satellite image during the 2011 Cordón-Caulle eruption in Chile”

Meteorological Modelling

Research lines include:

  • Mesoscale Numerical Weather Prediction (NWP).
  • Data assimilation and downscaling from mesoscale NWP models to local-scale.
  • High-resolution wind field modelling in complex terrains using CFD, with Alya.
  • Modelling of the atmospheric boundary layer including turbulence and thermal effects.

Wind energy

Numerical modelling of wind farms is a crucial aspect in terms of both wind farm design and management. Applications using ALYA Green for high-resolution wind field modelling include:

  • Modelling of on-shore and off-shore wind farms considering all aspects affecting surface layer atmospheric flows such as topographic variations, heterogeneities in the roughness of the terrain, and the downwind wake effects of rotors.
  • Modelling of wind turbines using actuator disks and the HERMESH method. This method allows efficient local mesh refinements.
  • Wind resource assessment.
  • Forecast of short-term wind farm power production.
  • Tailored modelling postprocess using GoogleEarth to facilitate visualization and standard data interchange.

Geosciences Applications (GA)

New hydrocarbon discoveries suggest that large reservoirs might lie in the Atlantic shelves of America and Africa, hidden under saline or basaltic bodies. In order to localise and retrieve these hydrocarbons, new imaging methods to explore these sub-salt areas are being developed, which will require supercomputers with a peak performance in the order of 10 Petaflops, requiring innovative computer architectures.

Inverse model using Full Waveform Inversion

The research focuses on the use of elastic and electromagnetic wave modelling and inversion to develop new imaging algorithms, and in the practical implementation of those algorithms on different computer architectures.

In recent years state-of-the-art seismic imaging tools were developed (Kaleidoscope project) and received international recognition and awards. These tools used acoustic wave equations, requiring computers of 100 TFlops peak performance, however improved solutions require the use of elastic waves, multiplying computational needs 50-fold. The capability of modelling elastic waves opens the possibility of a real full waveform inversion procedure, leading to ever more accurate models of the Earth's subsurface.

In addition, the group aims to investigate the inversion of electromagnetic waves to obtain images of the subsurface's resistivity, which is directly associated with the different reservoir's fluid contents (water, hydrocarbons). Finally, a joint elastic and electromagnetic inversion could further constrain the properties of the hydrocarbon reservoirs beyond the capabilities of seismic or electromagnetic methods alone.

The research is focused on solving 4 grand challenges in hydrocarbon exploration:

  • Use elastic wave equation for modelling large onshore exploration surveys.
  • Develop a full waveform inversion algorithm based on elastic waves.
  • Develop a geophysical inversion method for electromagnetic waves.
  • Couple the elastic and the electromagnetic inversion procedures to obtain a novel reservoir characterisation tool.

The final objective is to merge all developments in the BSIT geophysical imaging toolkit (

Data Pre & Post Processing (DPPP)

The DPPP team, comprising scientists, programmers, and visual communication and interaction experts, works on the visual representation of HPC simulation data for three main communicative situations:

  • Data exploration as a tool for scientists.
  • Outreach to the community at large.
  • Publication of results from a scientist to other experts in the field.

The team develops visual protocols to transmit quantitative information in the most efficient manner. The resulting images and videos are typically included in papers and presentations, used to create films that non-experts can understand and enjoy, or to help scientists explore and extract information from their data more efficiently.

Wind farm simulation: air around wind turbines

The team uses an in-house developed pipeline for post-processing large volumes of data. The pipeline relies on a battery of small tools and scripts that automatically:

  • Convert proprietary data formats into intermediate standard formats (e.g. convert 3D data into VTK, HDF5, or OpenVDB formats).
  • Post-process and filter data using script-controlled open source programs (ParaView, PartIO).
  • Convert processed data into standard CGI formats (for open source software like Blender or commercial like Maya).
  • Queue and administer parallel rendering in Marenostrum and Minotauro.

One of the biggest challenges faced by modern visualisation is Big Data treatment: how to display and allow the exploration of very large data sets, in this case produced by simulations.

To address this problem, a parallel data visualisation system was developed, tested, and implemented. In this setup, simulation data distributed across a cluster (using Hadoop) is coupled to Paraview, where user exploration is translated to queries using Hive and Impala that return results in an interactive manner.

This enables researchers to use tools they are familiar with, but connected to potentially huge sets of data.

Achievements and work during 2014:

The DPPP team collaborated with the international music festival Sónar to produce and showcase a general information visualisation webpage, A.Track.Tion (, around the topic of music. The web showcased an interactive visualisation of the popularity of music genres since 1954, as well as the influence between them. Apart from the visualisation and interaction design, the team also obtained and analysed the data behind the application, creating and using automatic web crawlers and parsers to gather genre information from unstructured text sources like Wikipedia.

A multiyear collaboration was initiated with the Life Sciences department for designing and producing a graphical interface for the PELE molecular dynamics simulation software.

Within the frame of the European CONSOLIDER project SyEC, in February 2014 the Team released the short film documentary “Supercomputers” (

Snapshot of the shortfilm documentary “Supercomputers”

Through high-end render visualisations and interviews the film explains the impact of high performance computing on science, technology, and society. "Supercomputers" received the following awards at international film festivals:

  • Winner, Academia Film Olomouc International Festival of Science Documentary Films (Cech Republic),
  • Winner, 2014 Ronda International Scientific Film Biennial (Spain),
  • Winner, Daroca&Prisión Film Festival (Spain)
  • Official Selection, Science Film Festival (Southeast Asia)
  • Winner, Festival Internacional de Cortometrajes de Torrelavega (Spain)

The audience reached through the festival screenings exceeds half a million a people.

The team also produced numerous short videos and static images for scientists of BSC-CNS and the RES for use in scientific publications and webpages.

Simulation of airflow in the upper human respiratory system
Personal tools