Package net.algart.arrays

Class StandardIODataFileModel

java.lang.Object
net.algart.arrays.AbstractDataFileModel
net.algart.arrays.StandardIODataFileModel
All Implemented Interfaces:
DataFileModel<File>
public class StandardIODataFileModel extends AbstractDataFileModel implements DataFileModel<File>

Alternative implementation of DataFileModel, that creates usual Java files, but emulates mapping via standard read/write operations. More precisely, this implementation is based on readAllBuffer(FileChannel, long, ByteBuffer) and writeAllBuffer(FileChannel, long, ByteBuffer) methods: see comments to them for more details.

The data files, returned by this class, creates buffer holders containing NIO byte buffers, which can be direct or non-direct — it depends on the argument directBuffers of the constructors, having such argument, or on the defaultDirectBuffers() value for other constructors. The map method in a data file allocates such byte buffer (ByteBuffer.allocateDirect(size) or ByteBuffer.allocate(size)) and loads the file fragment into it (readAllBuffer(FileChannel, long, ByteBuffer)). The unmap(true/false) and flush(true/false) methods of the created buffer holder writes all data back to file (writeAllBuffer(FileChannel, long, ByteBuffer)). Reading data is cached in free Java memory via WeakReference technique, if this class was created via StandardIODataFileModel() or some other constructor, having no cacheReading argument, of via or some constructor with the argument cacheReading=true. Writing data is not cached.

The close() method in data files, returned by this class, completely closes the disk file and releases all connected system resources.

The temporary files are created and deleted in the same way as in DefaultDataFileModel.

This data file model is usually not so efficient as the DefaultDataFileModel that is based on true file mapping operations, especially for large bank size. However, this model can provide better performance, than DefaultDataFileModel, in some applications, when you need sequential access to very large disk files, much larger than available RAM. Some operation systems can be not optimized for a case of very intensive mapping of very large files, which is the base of DefaultDataFileModel; at the same time, this data file model uses only simple input/output OS calls and can have more predictable behaviour.

In addition, this variant is absolutely stable on all platforms, when Java file mapping technique has a serious bug in Java 1.5 and 1.6: "(fc) "Cleaner terminated abnormally" error in simple mapping test". Right now, this bug is usually avoided in current implementation of DefaultDataFileModel, but there is a little risk to get unexpected IOError. If you need maximal stability with Java 1.5 or 1.6, and the performance is enough, you may choose this data file model.

This class is immutable and thread-safe: there are no ways to modify settings of the created instance.

Author:
Daniel Alievsky
See Also:

  • Constructor Details

    • StandardIODataFileModel

      public StandardIODataFileModel()
      Equivalent to new StandardIODataFileModel(null, 0, true, defaultDirectBuffers()).

    • StandardIODataFileModel

      public StandardIODataFileModel(boolean cacheReading, boolean directBuffers)
      Equivalent to new StandardIODataFileModel(null, 0, cacheReading, directBuffers).
      Parameters:
      cacheReading - whether reading data should be cached in free Java memory.
      directBuffers - whether the data files, created by this class, should allocate byte buffers for mapping by ByteBuffer.allocateDirect(size) (if this argument is true) or by ByteBuffer.allocate(size) (if this argument is false).

    • StandardIODataFileModel

      public StandardIODataFileModel(File tempPath)
      Equivalent to new StandardIODataFileModel(tempPath, 0, true, defaultDirectBuffers()).
      Parameters:
      tempPath - the path where new temporary files will be created by AbstractDataFileModel.createTemporaryFile(boolean) method or null if the default temporary-file directory is to be used.

    • StandardIODataFileModel

      public StandardIODataFileModel(File tempPath, boolean cacheReading)
      Equivalent to new StandardIODataFileModel(tempPath, 0, cacheReading, defaultDirectBuffers()).
      Parameters:
      tempPath - the path where new temporary files will be created by AbstractDataFileModel.createTemporaryFile(boolean) method or null if the default temporary-file directory is to be used.
      cacheReading - whether reading data should be cached in free Java memory.

    • StandardIODataFileModel

      public StandardIODataFileModel(File tempPath, boolean cacheReading, boolean directBuffers)
      Equivalent to new StandardIODataFileModel(tempPath, 0, cacheReading, directBuffers).
      Parameters:
      tempPath - the path where new temporary files will be created by AbstractDataFileModel.createTemporaryFile(boolean) method or null if the default temporary-file directory is to be used.
      cacheReading - whether reading data should be cached in free Java memory.
      directBuffers - whether the data files, created by this class, should allocate byte buffers for mapping by ByteBuffer.allocateDirect(size) (if this argument is true) or by ByteBuffer.allocate(size) (if this argument is false).

    • StandardIODataFileModel

      public StandardIODataFileModel(File tempPath, long prefixSize, boolean cacheReading, boolean directBuffers)
      Creates new instance of this class.

      Please see AbstractDataFileModel(File, long) about tempPath and prefixSize arguments.

      The data files, created by this model, will cache all loaded data in free Java memory (via WeakReference technique), if and only if cacheReading argument is true. In many cases, caching can improve performance. But if you are sure that the data will be usually read once, it is better to pass false as the constructor argument, because needless caching may increase disk swapping.

      The directBuffers argument defines the kind of byte buffers, which will be allocated in Java memory by data files, created by this class. If this argument is true, their map method will use direct byte buffers, i.e. will allocate them by ByteBuffer.allocateDirect(size) call. If this argument is false, the map method will use non-direct byte buffers, i.e. will allocate them by ByteBuffer.allocate(size) call.

      Note: the value directBuffers=false can slow down processing AlgART arrays, created with help of this data file model, if the element type is other than byte. The reason is that byte buffers, created by this model, are converted to the necessary element type by methods ByteBuffer.asShortBuffer, ByteBuffer.asIntBuffer(), ByteBuffer.asDoubleBuffer(), etc., and accessing to content of the AlgART arrays is performed via these views of the original byte buffers. In a case of direct byte buffers such conversion is usually performed at the low processor level in a native code, and access to their views like ByteBuffer.asDoubleBuffer() is performed with the maximal possible speed. Unlike this, in a case of non-direct byte buffers such conversion is performed in Java code, and access to each element of these views requires some calculations. For example, reading any element of ByteBuffer.asDoubleBuffer() means reading 8 bytes, joining them into a long by "<<" and "|" operators and calling Double.longBitsToDouble method — and these operations will be performed while any form of reading elements from the AlgART array DoubleArray. Thus, access to AlgART arrays, excepting ByteArray, becomes slower. The speed difference is usually not very large, but can be appreciable for simple applications with intensive accessing AlgART arrays.

      On the other hand, please note: the value directBuffers=true can increase requirements to memory and the risk of unexpected OutOfMemoryError, if you are using a lot of large banks, especially for 32-bit JVM. Direct byte buffers are usually allocated not in the usual Java heap, but in some separate address space. As a result, on 32-bit JVM the maximal amount of memory, which can be allocated in direct byte buffers, can be essentially less than -Xmx JVM argument, when -Xmx is 500–1000 MB or greater. If you need maximally stable application and you are planning to use all or almost all available memory, specified via -Xmx JVM argument, for banks of this data file model (to reduce disk swapping), you should prefer non-direct buffers (directBuffers=false). Of course, this problem should not occur while using default settings for recommendedNumberOfBanks() and recommendedBankSize(boolean) — 32 banks per 64 KB.

      Parameters:
      tempPath - the path where new temporary files will be created by AbstractDataFileModel.createTemporaryFile(boolean) method or null if the default temporary-file directory is to be used.
      prefixSize - the value returned by AbstractDataFileModel.recommendedPrefixSize() implementation in this class.
      cacheReading - whether reading data should be cached in free Java memory.
      directBuffers - whether the data files, created by this class, should allocate byte buffers for mapping by ByteBuffer.allocateDirect(size) (if this argument is true) or by ByteBuffer.allocate(size) (if this argument is false).
      See Also:

  • Method Details

    • defaultDirectBuffers

      public static boolean defaultDirectBuffers()
      Default directBuffer flag, used when this this class is instantiated by a constructor without directBuffer argument. More precisely, if there is the system property "net.algart.arrays.StandardIODataFileModel.directBuffers", containing "true" or "false" string (in lower case), this method returns true if this property contains "true" and false if this property contains "false" If there is no such property, or if it contains illegal string (not "true" or "false"), or if some exception occurred while calling System.getProperty, this method returns true (default value). The value of this system property is loaded and checked only once while initializing StandardIODataFileModel class.
      Returns:
      default directBuffer flag.

    • isDirectBuffers

      public final boolean isDirectBuffers()
      Returns the directBuffers argument, passed to the main constructor (or to other constructors, having such argument) while creating this instance. If this instance was created by constructors, which have no directBuffers, the result of this method is equal to defaultDirectBuffers() value.
      Returns:
      whether the data files, created by this class, should allocate byte buffers for mapping by ByteBuffer.allocateDirect(size) (if this flag is true) or by ByteBuffer.allocate(size) (if this flag is false).

    • getDataFile

      public DataFile getDataFile(File path, ByteOrder byteOrder)
      This implementation returns the data file corresponding to usual Java file new java.io.File(path), with DataFile.map method that use usual read/write operation instead of Java mapping.

      This method never throws java.io.IOError.

      Specified by:
      getDataFile in interface DataFileModel<File>
      Specified by:
      getDataFile in class AbstractDataFileModel
      Parameters:
      path - the path to disk file (as the argument of new java.io.File(path)).
      byteOrder - the byte order that will be always used for mapping this file.
      Returns:
      new instance of DataFile object.
      Throws:
      NullPointerException - if one of the passed arguments is null.

    • getPath

      public File getPath(DataFile dataFile)
      Returns the absolute path to the disk file (java.io.File.getAbsoluteFile()). The argument may be created by this data file model or by DefaultDataFileModel.

      This method never throws java.io.IOError.

      Specified by:
      getPath in interface DataFileModel<File>
      Specified by:
      getPath in class AbstractDataFileModel
      Parameters:
      dataFile - the data file.
      Returns:
      the absolute path to the disk file.
      Throws:
      NullPointerException - if the argument is null.
      ClassCastException - if the data file was created by data file model, other than DefaultDataFileModel or StandardIODataFileModel.

    • isAutoDeletionRequested

      public boolean isAutoDeletionRequested()

      This implementation returns true.

      Specified by:
      isAutoDeletionRequested in interface DataFileModel<File>
      Specified by:
      isAutoDeletionRequested in class AbstractDataFileModel
      Returns:
      true.

    • recommendedNumberOfBanks

      public int recommendedNumberOfBanks()

      This implementation returns the value Integer.getInteger("net.algart.arrays.StandardIODataFileModel.numberOfBanks", 32), stored while initializing this StandardIODataFileModel class, or default value 32 if some exception occurred while calling Integer.getInteger. If this value is less than 2, returns 2.

      Specified by:
      recommendedNumberOfBanks in interface DataFileModel<File>
      Specified by:
      recommendedNumberOfBanks in class AbstractDataFileModel
      Returns:
      the recommended number of memory banks.

    • recommendedBankSize

      public int recommendedBankSize(boolean unresizable)

      This implementation returns the value Integer.getInteger("net.algart.arrays.StandardIODataFileModel.bankSize",65536) (64 KB) (regardless of the argument), stored while initializing StandardIODataFileModel class, or default value 65536 if some exception occurred while calling Integer.getInteger. If this property contains invalid value (for example, not a power of two), this value is automatically corrected to the nearest valid one.

      Specified by:
      recommendedBankSize in interface DataFileModel<File>
      Specified by:
      recommendedBankSize in class AbstractDataFileModel
      Parameters:
      unresizable - ignored in this implementation.
      Returns:
      the recommended size of every memory bank in bytes.
      See Also:

    • temporaryFilePrefix

      public String temporaryFilePrefix()
      This implementation returns "stdmm";
      Overrides:
      temporaryFilePrefix in class AbstractDataFileModel
      Returns:
      "stdmm".

    • toString

      public String toString()
      Returns a brief string description of this class.

      The result of this method may depend on implementation.

      Overrides:
      toString in class Object
      Returns:
      a brief string description of this object.

    • readAllBuffer

      public static void readAllBuffer(FileChannel fileChannel, long position, ByteBuffer dest) throws IOException
      Reads all content of the byte buffer (dest.limit() bytes) from the given position of the file channel. Current positions in the file channel and byte buffer are ignored and not changed.

      Unlike FileChannel.read(ByteBuffer dst, long position), this method guarantees that all dest.limit() bytes will be really read from the file channel. (FileChannel.read method might not fill all buffer: it returns the number of successfully read bytes.) To provide this guarantee, this method performs a loop of calls of FileChannel.read method; if it cannot load all bytes, it throws EOFException.

      In a case of exception, the content of dest byte buffer can be partially changed (loaded from the file). The file channel position and byte buffer positions are not changed in any case.

      Parameters:
      fileChannel - the file channel.
      position - the file position at which the transfer is to begin; must be non-negative.
      dest - the destination byte buffer.
      Throws:
      IllegalArgumentException - if the position is negative.
      EOFException - if the byte buffer cannot be fully read.
      IOException - in the same situations as FileChannel.read(ByteBuffer dst) method.

    • writeAllBuffer

      public static void writeAllBuffer(FileChannel fileChannel, long position, ByteBuffer src) throws IOException
      Writes all content of the byte buffer (src.limit() bytes) to the given position of the file channel. Current positions in the file channel and byte buffer are ignored and not changed.

      Unlike FileChannel.write(ByteBuffer dst, long position), this method guarantees that all src.limit() bytes will be really written to the file channel. (FileChannel.write method might not write all buffer: it returns the number of successfully written bytes.) To provide this guarantee, this method performs a loop of calls of FileChannel.write method; if it cannot write all bytes, it throws IOException.

      In a case of exception, the content of the file can be partially changed (written from the byte buffer). The file channel position and byte buffer positions are not changed in any case.

      Parameters:
      fileChannel - the file channel.
      position - the file position at which the transfer is to begin; must be non-negative.
      src - the source byte buffer.
      Throws:
      IllegalArgumentException - if the position is negative.
      IOException - in the same situations as FileChannel.write(ByteBuffer dst) method, and also if the byte buffer cannot be fully written.