What is EZTrace ?
=============================================
EZTrace is a tool that aims at generating automatically execution traces
from HPC (High Performance Computing) programs. It generates execution
trace files that can be interpreted by visualization tools such as
ViTE.


Requirements
=============================================

In order to run EZTrace, you need the following software:
* autoconf 2.63

Optional:
* any MPI implementation. Make sure your MPI installation provides dynamic libraries.
* DynInst (you can download it here: http://www.dyninst.org/ )

We can also use a custom version of the following software. Those are already included in eztrace, but you can provide your own versions:
* FxT 0.2 or later: the latest release can be downloaded here: http://download.savannah.gnu.org/releases/fkt/
* gtg (http://gforge.inria.fr/projects/gtg/)


Getting EZTrace
=============================================

* You can get the latest stable release on EZTrace website:
  http://eztrace.gforge.inria.fr/

* Current development version is available via svn
  svn checkout svn+ssh://scm.gforge.inria.fr/svn/eztrace

After getting the latest development version (from SVN), you need to run './bootstrap'

Building EZTrace
=============================================

First, you need to configure EZTrace by invoking the configure script.

$ ./configure --prefix=<WHERE_YOU_INSTALL_EZTRACE>
Options to configure

You can pass several options to the configure script for specifying where it should find the needed libraries:
--with-fxt=$FXT_ROOT : specify where FxT is installed.
--with-gtg=$GTG_ROOT : specify where GTG is installed.
--with-mpi=$MPI_ROOT : specify where MPI is installed. The mpi.h file must be located in the $MPI_ROOT/include/ directory.
--with-mpi-include=<PATH_TO_MPI.H> : specify the directory that contains the mpi.h file.
--with-dyninst=$DYNINST_ROOT : specify where DynInst is installed.

Once EZTrace is configured, just type:

$ make
$ make install


Using EZTrace
=============================================

Choosing the functions to instrument
------------------------------------

First, you should select the functions you want to instrument in your application. For this, you can
set the EZTRACE_TRACE environment variable to the list of plugins that should be used:

$ export EZTRACE_TRACE="pthread mpi"

You can get the list of available plugins with the eztrace_avail command:

$ eztrace_avail
3       stdio   Module for stdio functions (read, write, select, poll, etc.)
2       pthread Module for PThread synchronization functions (mutex, semaphore, spinlock, etc.)
6       papi    Module for PAPI Performance counters
1       omp     Module for OpenMP parallel regions
4       mpi     Module for MPI functions
5       memory  Module for memory functions (malloc, free, etc.)

You can get the list of selected modules with the eztrace_loaded command:
$ export EZTRACE_TRACE="pthread mpi"
$ eztrace_avail
2       pthread Module for PThread synchronization functions (mutex, semaphore, spinlock, etc.)
4       mpi     Module for MPI functions


Creating execution traces
-------------------------

Once the list of plugins is selected, you can run your application with eztrace. For example:

$ eztrace ./my_program my_arg1 my_arg2

This command line executes your program and generates a trace file in the /tmp directory
(usually the file is named /tmp/<username>_eztrace_log_rank_#rank).


Using EZTrace with MPI
----------------------

EZTrace only needs to set a LD_PRELOAD variable for each MPI process. You can specify to mpiexec the
value of LD_PRELOAD using the eztrace -e command. For example, you can run the following command:

$ mpirun -np 16 `eztrace -e my_app`

When your application ends. Each process writes a file named /tmp/<username>_eztrace_log_rank_<rank>.


Changing The output directory
-----------------------------

By default, each process saves its trace in the local /tmp directory. You can change this by
setting EZTRACE_TRACE_DIR.


Merging execution traces
------------------------

Once the execution traces are recorded, you can merge and convert them into a file format that
can be read by your visualization software:

$ eztrace_convert -o my_paje.trace /tmp/<username>_eztrace_log_rank_0 /tmp/<username>_eztrace_log_rank_1

This converts the trace files into the Paje format. If gtg is installed with OTF support (this is enabled by default), you can choose to convert into the OTF file format with the -t OTF option :

$ eztrace_convert -t OTF /tmp/<username>_eztrace_log_rank_0 /tmp/<username>_eztrace_log_rank_1


Filtering events
----------------
You can select the plugins to use for the conversion phase by using the EZTRACE_TRACE variable: if your traces contains MPI
and OpenMP events, setting EZTRACE_TRACE to mpi and calling eztrace_convert creates an output trace that only contains mpi events.


Computing statistics
--------------------

Instead of creating a merged trace file, you can tell EZTrace to compute statistics on the recorded traces:

$ eztrace_stats /tmp/<username>_eztrace_log_rank_0 /tmp/<username>_eztrace_log_rank_1
[...]
PThread:
-------
6 locks acquired

MPI:
---
27 messages sent
MPI_RECV :10 calls
MPI_BARRIER :11 calls
163 events handled



Defining custom plugins
------------------------

Since EZTrace works with plugins, you can create one and instrument the functions that you want.
An example of plugin is available in the example directory.

Once your plugin is created, you should tell EZTrace where to find it. For this, just set the
EZTRACE_LIBRARY_PATH to the appropriate director(ies)y:

$ export EZTRACE_LIBRARY_PATH=plugin1:plugin2
$ eztrace_avail
  3       stdio   Module for stdio functions (read, write, select, poll, etc.)
  2       pthread Module for PThread synchronization functions (mutex, semaphore, spinlock, etc.)
  6       papi    Module for PAPI Performance counters
  1       omp     Module for OpenMP parallel regions
  4       mpi     Module for MPI functions
  5       memory  Module for memory functions (malloc, free, etc.)
  99 plugin1 Example module for libplugin1
  98 plugin2 Example module for the plugin2 library


Generating custom plugins
--------------------------

You can generate one plugin and instrument the functions you want to.
In order to generate your plugin, you need to create a file containing:

* The name of the library you want to trace (libNAME.so)
* A brief description of the library (optional)
* An ID to identifie the module (0? is reserved for eztrace internal use. Thus you can use any between 10 and ff)
* The prototype of the functions you want to instrument


Basically, the file should look like that:

BEGIN_MODULE
NAME example_lib
DESC "module for the example library"
ID 99
int example_do_event(int n)
double example_function1(double* array, int array_size)
END_MODULE

Now use eztrace_create_plugin to generate the plugin source code:

$ eztrace_create_plugin example.tpl
New Module
Module name : 'example_lib'
Module description : '"module for the example library"'
Module id : '99'
        emulate record_state for 'example_do_event'
Function 'example_do_event' done
        emulate record_state for 'example_function1'
Function 'example_function1' done
End of Module example_lib

The source code is generated in the output directory. Just type:

$ make

Now set the EZTRACE_LIBRARY_PATH to the appropriate directory and you
	are good to go.

You can also specify (in the example.tpl file) the way a fonction
  is depicted in the output trace. For instance:

int submit_job(int* array, int array_size)
BEGIN
 ADD_VAR("job counter", 1)
END

Specifies that when the submit_job function is called, the output
trace should increment the "job counter" variable. You can now track
the value of a variable!

The test/module_generator directory contains several scripts that
demonstrate the various commands available.


Environment variables
=====================
Here is a list of the environment variables that you can use for tuning EZTrace.

* General-purpose variables:
  - EZTRACE_TRACE_DIR: specifies the directory in which the traces are created
  - EZTRACE_LIBRARY_PATH: specifies the directories in which eztrace can find eztrace modules
  - EZTRACE_TRACE: specifies the list of eztrace modules to load
  - EZTRACE_BUFFER_SIZE: specifies the size of the buffer in which eztrace strores events.
* Error-handling variables
  - EZTRACE_NO_SIGNAL_HANDLER: disables eztrace signal handling
* Hardware counters-related variables:
  - EZTRACE_PAPI_COUNTERS: selects the hardware counters to trace.
    example: export EZTRACE_PAPI_COUNTERS="PAPI_L3_TCM PAPI_FP_INS"
* MPI-related variables:
  - EZTRACE_MPI_DUMP_MESSAGES: tells EZTrace to dump the list of messages into a file. You can then compute your own statistics on MPI messages.


Known limitations
=================
  *  EZTrace relies on dlsym for interception various function
     calls. Thus, it only works with dynamically linked libraries.

  *  If EZTrace is compiled with a particular MPI implementation
     (OpenMPI for instance) it won't work if you run your application
     with another (MPICH2 for instance). So make sure your application
     uses the same MPI implementation as EZTrace.


Frequently A Questions
=============================================

Q: When I run my MPI application with EZTrace, all the processes generate /tmp/<username>_eztrace_log_rank_1
files. What's going wrong ?

This happens when EZTrace fails to intercept calls to MPI_Init (or MPI_Init_thread). There may be several
reasons for that:
* EZTrace MPI module was not compiled. For intercepting calls to MPI functions, you need the MPI module in your
installation (look for the $EZTRACE_ROOT/lib/libeztrace-autostart-mpi.so file). If you don't, it means that
something went wrong during the configuration of EZTrace, so check for errors or warnings in you config.log file.
* You specified the list of modules to use and the MPI module wasn't in there. Check your
EZTRACE_TRACE variable.
* Your application is statically linked against mpi. You can check this by calling ldd:
$ ldd my_application
[...]
        libmpi.so.0 => /[...]/lib/libmpi.so.0 (0x00639000)
[...]
(the name of the library may change depending on your MPI implementation: libmpi, libmpich, etc.)

If you still experience problems, please contact EZTrace development team so that we can fix your problem.

Q: What if I don't want to trace my whole application, but only a small part of it ?

Then, you can call eztrace_start() and eztrace_stop() in your code to specify the part to
be traced. You will need to #include <eztrace.h> and to link with libeztrace. Then, just
run your application as usual (ie. $ ./my_program my_arg1)


Q: I need to trace my program while using gdb, how can I do that ?

Just add the -d option to eztrace to enable gdb:
$ eztrace -d ./my_program my_arg1 my_arg2
Please note that this should be used only when a bug only happens while using EZTrace.


Q: I want my trace to be saved in a specific directory, how can I do that ?

By default, EZTrace saves the trace in the /tmp directory. You can change this by setting EZTRACE_TRACE_DIR.


Q: What if I don't care about OpenMP and I only want to see MPI communication ?

You can set EZTRACE_TRACE to the list of "modules" you want to activate. By default, all the available
modules are enabled, but you can tell EZTrace to trace only MPI or OpenMP functions:

$ export EZTRACE_TRACE=mpi
$ export EZTRACE_TRACE=omp
$ export EZTRACE_TRACE="omp mpi"


Q: Can EZTrace generate an OTF trace file so that I can visualize it with Vampir ?

Yes, since EZTrace relies on GTG for writing output traces, it can generate OTF trace files.
When converting the trace with eztrace_convert, just add the -t OTF option:

$ eztrace_convert -t OTF /tmp/<username>_eztrace_log_rank_0 /tmp/<username>_eztrace_log_rank_1



Troubleshooting
=============================================

If you encounter a bug or want some explanation about EZTrace, feel
free to ask to the developers:

* On the developer mailing list
* On our IRC channel:
  - server: chat.freenode.net
  - channel: #eztrace

