JSC Homepage
	JUBL Home - IBM Blue Gene/L
	Configuration
	User Info
	Computing Resources Job Accounting Logon Sample Scheduling Policy LoadLeveler for BG/L Checkpointing Programming Environment Unsupported Features llview for BG/L scalasca on BG/L Memory Limits on JUBL FAQ
	Software
	Documentation
	Links

User Info scalasca on BG/L

Using scalasca on JUBL

Access

SCALASCA versions are installed as modules on the IBM Blue Gene/L system JUBL (in /usr/local/modules):

% module avail
------------ /usr/local/modules/modulefiles ------------
scalasca/0.5(default)
% module whatis scalasca
----------- /usr/local/modules/modulefiles -------------
scalasca/0.5: Scalable automatic performance analysis toolset
% module help scalasca
-------------------------------------------------------------------
Module Specific Help for /usr/local/modules/modulefiles/scalasca/0.5:

SCALASCA 0.5
Scalable automatic performance analysis of large-scale applications
toolset comprising:
- EPIK measurement system
- SCOUT parallel automatic event trace analysis tool
- CUBE 3.0 performance data presentation tool

http://www.scalasca.org/

Includes a version of EXPERT sequential event analysis tool
- using EPIK & CUBE incompatible with KOJAK 2.2 release

http://www2.fz-juelich.de/zam/kojak/

Usage:
- Automatic instrumentation by adding kinst to compiler/linker
variables in your Makefiles, e.g., CC="kinst bglrts_xlc"
- Semi-automatic instrumentation by manual instrumentation of
important functions/regions with POMP directives and adding
kinst-pomp to compiler/linker variables in your Makefiles,
e.g., CC="kinst-pomp bglrts_xlc"
- (Re-)compile your application
- Execute your application
- Execute scout parallel analyzer on same number of processors
as application with mpirun as "-exe $SCALASCA_RTS/scout"
- View results with cube3

See also: $SCALASCA_DIR/doc/USAGE
-------------------------------------------------------------------
% module load scalasca
% module list
Currently Loaded Modulefiles:
1) scalasca/0.5

Versions

SCALASCA 0.5 is a technology preview for testing and evaluation purposes. It includes the latest development version of KOJAK which is incompatible with the KOJAK 2.2 release (from Nov 2006). Additional versions may become available in future.

Each version includes the SCALASCA/KOJAK instrumenter, EPIK measurement system, SCOUT trace analyzer and CUBE3 analysis browser (and associated tools). (Don't mix the use of components from different SCALASCA and KOJAK versions, as file formats may be incompatible.)

The SCALASCA 0.5 installation is configured with MPICH and the IBM cross-compilers for the BG/L compute nodes (blrts_xlc, blrts_xlC, blrts_xlf): the local compiler wrappers can also be used (mpcc, mpCC, mpxlf). GNU compiler collection compilers are not explicitly supported by this installation. When preparing your application with SCALASCA (kinst or kinst-pomp), the compilers' flags should be properly recognised, and automatic user function instrumentation can be enabled (kinst).

PAPI is configured so that hardware counter metric measurements can be included in experiments, however, such metrics are ignored by the SCOUT analyser. (MPI2 RMA/1-sided communication tracing is not supported by these installations.)

Issues

Support is only provided for measuring and analysing applications running on the compute nodes launched by mpirun: instrumentation and analysis browsing are (only) supported on the front-end nodes. SCOUT analysis reports (epik.cube files) can be extracted from experiment archives and browsed on any system where CUBE3 is installed (e.g., JUMP or a workstation/notebook computer).

A number of variables can be used to control the EPIK measurement runtime configuration of an instrumented executable: for an annotated list of EPIK configuration variables, and their current settings, run the epk_conf command. Variables can be specified via environment variables or in a configuration file called "EPIK.CONF": by default the current directory is searched for this file, or an alternative location can be specified with the EPK_CONF environment variable.

For (large-scale) MPI applications ensure that the EPK_LDIR and EPK_GDIR variables are set to the same location, as this avoids intermediate file writing and can greatly improve performance.

Due to the restricted memory on BG/L compute nodes, care is required when selecting an appropriate buffer size for traces. The default trace buffer size (ELG_BUFFER_SIZE) for each process is rather small and typically only adequate for very short traces. It is therefore recommended to set the trace buffer size as large as available memory permits: if too large a size is specified, the application will be unable to run or fail to acquire memory.

When using kinst automatic user function instrumentation, ELG_BLACKLIST can be used to specify a file containing the names of functions (one per line) to be excluded from measurement collection, thereby reducing measurement perturbation and trace size. Although convenient for experimentation, the remaining instrumentation handling overhead can still be significant and in such cases it is preferable to compile such functions/modules separately without kinst instrumentation.

SCOUT analysis requires exactly the same number of MPI processes as measured in an experiment: typically it will be convenient to launch SCOUT immediately following the measurement in a single batch script so that mpirun can be configured with the same number of processes and run on the same partition for both steps.

The SCOUT analyser for the compute nodes should typically be specified to mpirun as “-exe $SCALASCA_RTS/scout” with commandline arguments and environment variables specified in the usual manner: e.g., -arg “epik_a” or -env “EPK_TITLE=a”.

Measurement experiments containing traces that are too large for SCOUT to analyse in virtual node (VN) mode may be analysable with the additional memory available in co-processor (CO) mode. SCOUT can typically analyse experiments containing process traces up to around 100MB in VN-mode and around 200MB in CO-mode. (Experiment archives are portable to other systems where sufficient processors with additional memory are available and a compatible version of SCALASCA/SCOUT is installed, however, the size of experiment archives typically prohibits this.)

Commands, library interfaces and file formats are subject to change (unstable).

Reference

http://www.scalasca.org/ – SCALASCA project website, including overview of toolset architecture, presentations and publications, and software download.

http://www2.fz-juelich.de/zam/kojak/ – KOJAK project website, including overview of toolset architecture, presentations and publications, and software download.

$SCALASCA_DIR/doc – Directory containing documentation for the SCALASCA installation, including USAGE notes (covering application program instrumentation, runtime measurement control, and automatic and manual analyses), OPEN_ISSUES documenting open issues, known limitations and unimplemented functionality, performance property patterns used by the SCOUT automatic trace analyser, and CUBE3 user manual for the analysis report browser.

$SCALASCA_DIR/example – Directory containing example Fortran and C programs with different forms of instrumentation, and an example experiment archive.

scalasca@scalasca.org – Mailing list for SCALASCA comments, questions and bug reports.

---

last change 12.01.2007 | Brian Wylie | Print