Package org.apache.lucene.store

Class NRTCachingDirectory

  • All Implemented Interfaces:
    java.io.Closeable, java.lang.AutoCloseable
    public class NRTCachingDirectory
extends Directory
    Wraps a RAMDirectory around any provided delegate directory, to be used during NRT search.

    This class is likely only useful in a near-real-time context, where indexing rate is lowish but reopen rate is highish, resulting in many tiny files being written. This directory keeps such segments (as well as the segments produced by merging them, as long as they are small enough), in RAM.

    This is safe to use: when your app calls {IndexWriter#commit}, all cached files will be flushed from the cached and sync'd.

    Here's a simple example usage: 

       Directory fsDir = FSDirectory.open(new File("/path/to/index"));
   NRTCachingDirectory cachedFSDir = new NRTCachingDirectory(fsDir, 5.0, 60.0);
   IndexWriterConfig conf = new IndexWriterConfig(Version.LUCENE_47, analyzer);
   IndexWriter writer = new IndexWriter(cachedFSDir, conf);

    This will cache all newly flushed segments, all merges whose expected segment size is <= 5 MB, unless the net cached bytes exceeds 60 MB at which point all writes will not be cached (until the net bytes falls below 60 MB).

    • Constructor Summary

      Constructors 
      Constructor Description
      NRTCachingDirectory​(Directory delegate, double maxMergeSizeMB, double maxCachedMB)
      We will cache a newly created output if 1) it's a flush or a merge and the estimated size of the merged segment is <= maxMergeSizeMB, and 2) the total cached bytes is <= maxCachedMB

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void clearLock​(java.lang.String name)
      Attempt to clear (forcefully unlock and remove) the specified lock.
      void close()
      Close this directory, which flushes any cached files to the delegate and then closes the delegate.
      IndexOutput createOutput​(java.lang.String name, IOContext context)
      Creates a new, empty file in the directory with the given name.
      Directory.IndexInputSlicer createSlicer​(java.lang.String name, IOContext context)
      Creates an Directory.IndexInputSlicer for the given file name.
      void deleteFile​(java.lang.String name)
      Removes an existing file in the directory.
      boolean fileExists​(java.lang.String name)
      Returns true iff a file with the given name exists.
      long fileLength​(java.lang.String name)
      Returns the length of a file in the directory.
      Directory getDelegate()  
      LockFactory getLockFactory()
      Get the LockFactory that this Directory instance is using for its locking implementation.
      java.lang.String getLockID()
      Return a string identifier that uniquely differentiates this Directory instance from other Directory instances.
      java.lang.String[] listAll()
      Returns an array of strings, one for each file in the directory.
      java.lang.String[] listCachedFiles()  
      Lock makeLock​(java.lang.String name)
      Construct a Lock.
      IndexInput openInput​(java.lang.String name, IOContext context)
      Returns a stream reading an existing file, with the specified read buffer size.
      void setLockFactory​(LockFactory lf)
      Set the LockFactory that this Directory instance should use for its locking implementation.
      long sizeInBytes()
      Returns how many bytes are being used by the RAMDirectory cache
      void sync​(java.util.Collection<java.lang.String> fileNames)
      Ensure that any writes to these files are moved to stable storage.
      java.lang.String toString()  

      • Methods inherited from class org.apache.lucene.store.Directory

        copy

      • Methods inherited from class java.lang.Object

        equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

    • Constructor Detail

      • NRTCachingDirectory

        public NRTCachingDirectory​(Directory delegate,
                           double maxMergeSizeMB,
                           double maxCachedMB)
        We will cache a newly created output if 1) it's a flush or a merge and the estimated size of the merged segment is <= maxMergeSizeMB, and 2) the total cached bytes is <= maxCachedMB

    • Method Detail

      • getDelegate

        public Directory getDelegate()

      • getLockFactory

        public LockFactory getLockFactory()
        Description copied from class: Directory
        Get the LockFactory that this Directory instance is using for its locking implementation. Note that this may be null for Directory implementations that provide their own locking implementation.
        Specified by:
        getLockFactory in class Directory

      • setLockFactory

        public void setLockFactory​(LockFactory lf)
                    throws java.io.IOException
        Description copied from class: Directory
        Set the LockFactory that this Directory instance should use for its locking implementation. Each * instance of LockFactory should only be used for one directory (ie, do not share a single instance across multiple Directories).
        Specified by:
        setLockFactory in class Directory
        Parameters:
        lf - instance of LockFactory.
        Throws:
        java.io.IOException

      • getLockID

        public java.lang.String getLockID()
        Description copied from class: Directory
        Return a string identifier that uniquely differentiates this Directory instance from other Directory instances. This ID should be the same if two Directory instances (even in different JVMs and/or on different machines) are considered "the same index". This is how locking "scopes" to the right index.
        Overrides:
        getLockID in class Directory

      • makeLock

        public Lock makeLock​(java.lang.String name)
        Description copied from class: Directory
        Construct a Lock.
        Specified by:
        makeLock in class Directory
        Parameters:
        name - the name of the lock file

      • clearLock

        public void clearLock​(java.lang.String name)
               throws java.io.IOException
        Description copied from class: Directory
        Attempt to clear (forcefully unlock and remove) the specified lock. Only call this at a time when you are certain this lock is no longer in use.
        Specified by:
        clearLock in class Directory
        Parameters:
        name - name of the lock to be cleared.
        Throws:
        java.io.IOException

      • toString

        public java.lang.String toString()
        Overrides:
        toString in class Directory

      • sizeInBytes

        public long sizeInBytes()
        Returns how many bytes are being used by the RAMDirectory cache

      • fileExists

        public boolean fileExists​(java.lang.String name)
                   throws java.io.IOException
        Description copied from class: Directory
        Returns true iff a file with the given name exists.
        Specified by:
        fileExists in class Directory
        Throws:
        java.io.IOException

      • deleteFile

        public void deleteFile​(java.lang.String name)
                throws java.io.IOException
        Description copied from class: Directory
        Removes an existing file in the directory.
        Specified by:
        deleteFile in class Directory
        Throws:
        java.io.IOException

      • fileLength

        public long fileLength​(java.lang.String name)
                throws java.io.IOException
        Description copied from class: Directory
        Returns the length of a file in the directory. This method follows the following contract:
        • Must throw FileNotFoundException if the file does not exist (not java.nio.file.NoSuchFileException of Java 7).
        • Returns a value ≥0 if the file exists, which specifies its length.
        Specified by:
        fileLength in class Directory
        Parameters:
        name - the name of the file for which to return the length.
        Throws:
        java.io.IOException - if there was an IO error while retrieving the file's length.

      • listCachedFiles

        public java.lang.String[] listCachedFiles()

      • createOutput

        public IndexOutput createOutput​(java.lang.String name,
                                IOContext context)
                         throws java.io.IOException
        Description copied from class: Directory
        Creates a new, empty file in the directory with the given name. Returns a stream writing this file.
        Specified by:
        createOutput in class Directory
        Throws:
        java.io.IOException

      • sync

        public void sync​(java.util.Collection<java.lang.String> fileNames)
          throws java.io.IOException
        Description copied from class: Directory
        Ensure that any writes to these files are moved to stable storage. Lucene uses this to properly commit changes to the index, to prevent a machine/OS crash from corrupting the index.

        NOTE: Clients may call this method for same files over and over again, so some impls might optimize for that. For other impls the operation can be a noop, for various reasons.
        Specified by:
        sync in class Directory
        Throws:
        java.io.IOException

      • openInput

        public IndexInput openInput​(java.lang.String name,
                            IOContext context)
                     throws java.io.IOException
        Description copied from class: Directory
        Returns a stream reading an existing file, with the specified read buffer size. The particular Directory implementation may ignore the buffer size. Currently the only Directory implementations that respect this parameter are FSDirectory and CompoundFileDirectory.
      • Must throw FileNotFoundException if the file does not exist (not java.nio.file.NoSuchFileException of Java 7).
Specified by:
openInput in class Directory
Throws:
java.io.IOException
Overrides:
createSlicer in class Directory
Throws:
java.io.IOException - if an IOException occurs