SIONlib: Scalable I/O library for parallel access to task-local files

Templates for parallel I/O with SIONlib
Sample parallel MPI program for I/O with SIONlib
Serial program for SIONlib file access
Other examples

Templates for parallel I/O with sionlib

All templates can also be found in the distribution package under the PATH: sionlib/examples

Parallel write: 
sid=sion_paropen_mpi( ... ,chunksize, comm, &fileptr, ...)  # collective

loop: { 
        sion_ensure_free_space(sid,nbytes);                 # non-collective
        fwrite(data,1,nbytes,fileptr)
      }

sion_parclose_mpi(sid);                                     # collective

- the call to sion_ensure_free_space can be omitted, if it is guaranteed that not more bytes as specified in chunksize are written to the file.
- chunksize can be different on different tasks
- if the data does not fit in the current chunk, sion_ensure_free_space would assign a new chunk in the file to the task and also advance the file pointer to the new position. This operation is non-collective; all information about the locally used chunks and the number of bytes written to the chunks are buffered in memory and stored in the sion file in the collective sion_parclose_mpi.
- one parameter of the open function is an MPI communicator, which allows opening a sion file from a subset of all MPI tasks


Parallel read: 
sid=sion_paropen_mpi( ... ,&chunksize, comm, &fileptr, ...) # collective

while((!sion_feof(sid))) {                                  # non-collective
      btoread=sion_bytes_avail_in_block(sid);               # non-collective
      bread=fread(localbuffer,1,btoread,fileptr);     
}             

sion_parclose_mpi(sid);                                     # collective

- it must be ensured that the number of bytes read from a chunk is not larger as the number of bytes written to it. For this the function sion_bytes_avail_in_block provides this number for the current chunk.

- if all bytes of a chunk are already read and there are more chunks available for this task, sion_feof will advance the filepointer to the start position of the next chunk in the sion file

- sion_paropen_mpi is collective. The meta data will read only by one task and broadcasted to all other tasks. It is also possible to open/close the sion file on each with serial sion functions without collective operations, see function sion_open_rank.


Parallel read without collective operation: 
sid=sion_open_rank( ... ,&chunksize, rank, &fileptr, ...)   # non-collective

while((!sion_feof(sid))) {                                  # non-collective
      btoread=sion_bytes_avail_in_block(sid);               # non-collective
      bread=fread(localbuffer,1,btoread,fileptr);     
}             

sion_close(sid);                                            # non-collective

- the meta-information of the sion file will be read from each task. This is a parallel access to same filesystem blocks, however this are only read accesses which should not be lock the filesystem block.

top of page

Sample parallel MPI program for I/O with SIONlib

This simple parallel program can be found under the PATH: sionlib/examples/simple
This directory contain also Makefiles to build the program. 
...
#include "sion.h"
#define FNAMELEN 255
#define BUFSIZE (1024*1024)

int main(int argc, char **argv)
{

  int            rank, size, globalrank, sid, i, numFiles;
  char          fname[FNAMELEN], *newfname=NULL;
  int           ;
  MPI_Comm      gComm, lComm;
  sion_int64    chunksize,left;
  sion_int32    fsblksize;
  size_t        btoread, bread, bwrote;
  char          *localbuffer;
  FILE          *fileptr;

  MPI_Init(&argc, &argv);
  MPI_Comm_size(MPI_COMM_WORLD, &size);
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);

  /* allocate and initalize a buffer  */
  localbuffer = (char *) malloc(BUFSIZE);
  srand(time(NULL));
  for (i = 0; i < BUFSIZE; i++) localbuffer[i] = (char) rand() % 256;

  /* inital parameters */
  strcpy(fname, "parfile.sion");
  numFiles   = 1;  
  gComm=lComm= MPI_COMM_WORLD;  
  chunksize  = 10*1024*1024;  
  fsblksize  = 1*1024*1024; 
  globalrank = rank;

  /* write */
  sid = sion_paropen_mpi(fname, "bw", &numFiles, gComm, &lComm, 
                         &chunksize, &fsblksize, &globalrank, &fileptr, &newfname);
  left=BUFSIZE;
  while (left > 0) {
    sion_ensure_free_space(sid, left);
    bwrote = fwrite(localbuffer, 1, left, fileptr);
    left -= bwrote;
  }
  sion_parclose_mpi(sid);  

  printf("Task %02d: wrote sionfile -> %s\n",rank,newfname);


  /* read */
  sid=sion_paropen_mpi(fname,"br",&numFiles,MPI_COMM_WORLD,&lComm,
                       &chunksize,&fsblksize, &globalrank, &fileptr, &newfname);
  while((!sion_feof(sid))) { 
    btoread=sion_bytes_avail_in_block(sid);         
    bread=fread(localbuffer,1,btoread,fileptr);     
  }             
  sion_parclose_mpi(sid);  

  printf("Task %02d: read sionfile -> %s\n",rank,newfname);

  MPI_Finalize();

  return (0);
}
top of page

Serial program for SIONlib file access

Templates for accessing SION-files from a serial program are shown below. The templates can also be found in the distribution package under the PATH: sionlib/examples


Serial write: 
sid=sion_open( ...,chunksize, &fileptr)

rank_loop: {
    sion_seek(sid,rank,SION_CURRENT_BLK,SION_CURRENT_POS);

    sion_ensure_free_space(sid,nbytes);
    fwrite(...,fileptr)
}
sion_close(id);

Serial read: 
sid=sion_open( ...,chunksize, &fileptr)                   
        
sion_get_locations(sid,&size,&blocks,&globalskip,&start_of_varheader,
                       &sion_localsizes,&sion_globalranks,
                       &sion_chunkcount,&sion_chunksizes);

loop: { 
   sion_seek(sid,rank,blknr,pos);
         
   fread(...,fileptr) 
}
sion_close(id);

- access to all ranks and chunks is possible (sion_seek)
- sion_get_locations returns pointers to internal fields, containing the number of chunks written by each task (sion_chunkcount) and their sizes (sion_chunksizes)

top of page

Other examples

The example directory also contains in sub-directory ./examples/pepc an example application dependant converter program for the parallel simulation application PEPC (Multi-Purpose Parallel Tree-Code, http://www2.fz-juelich.de/jsc/pepc/). This serial program converts data files between the native PEPC ASCII format and a binary format using sionfiles and can be used to build converter programs for own applications.

top of page

last change 28.01.2009 | Wolfgang Frings | Print