Extending the capability of performance tools to deal with the larger and larger machines being deployed is necessary in order to understand their actual behaviour and identify how to achieve performance expectations in the frequent case these are not met at a first try.
Trace based tools such as Paraver provide extremely powerful and flexible analysis capabilities to identify performance problems not detectable by profile based tools.
Scaling up the usability of trace based tools requires new techniques in both the acquisition and visualization phases. The CEPBA-tools approach distributes the functionalities required to tackle large systems in three different levels. Different acquisition techniques are used in the instrumentation package to control the data captured and maximize the ratio information to file size. Even so, huge traces may result. An intermediate level set of tools are used to summarize the generated Paraver traces into smaller traces, with the same format, but where some of the information has been summarized. Examples of filter functionalities at this level include summarization of certain events in periodic software counters, selection of specific time intervals or events or combinations. The final component of the environment is Paraver, the visualizer itself. Different rendering techniques have been introduced in Paraver to visualize traces of many processes while still being able to convey to the analyst the information relevant to identify problems at very coarse level as well as the capabilities to dig down to very detailed levels. The paper describes in detail the techniques being used along those lines in the CEPBA-tools environment in order to support the anaysis of applications in the MareNostrum platform.