Package it.unimi.di.big.mg4j.index

Class MultiTermIndexIterator

java.lang.Object

it.unimi.di.big.mg4j.search.AbstractDocumentIterator

it.unimi.di.big.mg4j.search.AbstractIntervalDocumentIterator

it.unimi.di.big.mg4j.search.AbstractCompositeDocumentIterator

it.unimi.di.big.mg4j.search.AbstractUnionDocumentIterator

it.unimi.di.big.mg4j.index.MultiTermIndexIterator

All Implemented Interfaces:

IndexIterator, DocumentIterator

public class MultiTermIndexIterator
extends AbstractUnionDocumentIterator
implements IndexIterator

A virtual index iterator that merges several component index iterators.

This class adds to AbstractUnionDocumentIterator an interval iterator generating the OR of the intervals returned for each of the documents involved. The main difference with an OrDocumentIterator built on the same array of component iterators is that this class implements IndexIterator and hence provides a count() (the sum of counts of those component iterators positioned on the current document), frequency() and nextPosition(). The frequency is by default the maximum frequency of a component iterator, but it can be set at construction time.

The main raison d'être of this class is support for query expansion: a blind application of OrDocumentIterator to an array of index iterators would mislead scorers such as BM25Scorer because low-frequency terms (e.g., hapax legomena) would be responsible for most of the score.

Note that DocumentIteratorVisitor has a visit method for generic index iterator and a visit method for instances of this class. This approach provides additional flexibility—a scorer, for instance, might treat an instance of this class as a standard IndexIterator, or it might choose to query which terms actually appear and do something more sophisticated (for instance, using weights).

Nested Class Summary

Nested classes/interfaces inherited from class it.unimi.di.big.mg4j.search.AbstractUnionDocumentIterator

AbstractUnionDocumentIterator.IntHeapSemiIndirectPriorityQueue, AbstractUnionDocumentIterator.LongHeapSemiIndirectPriorityQueue

Nested classes/interfaces inherited from class it.unimi.di.big.mg4j.search.AbstractCompositeDocumentIterator

AbstractCompositeDocumentIterator.AbstractCompositeIndexIntervalIterator, AbstractCompositeDocumentIterator.AbstractCompositeIntervalIterator

Field Summary

Fields

Modifier and Type	Field	Description
protected int	id	The id of this iterator.
protected String	term	The term of this iterator.

Fields inherited from class it.unimi.di.big.mg4j.search.AbstractUnionDocumentIterator

front, frontSize, queue, refArray

Fields inherited from class it.unimi.di.big.mg4j.search.AbstractCompositeDocumentIterator

documentIterator, indexIterator, indexIteratorsWithoutPositions, n

Fields inherited from class it.unimi.di.big.mg4j.search.AbstractIntervalDocumentIterator

currentIterators, indices, intervalIterators, soleIndex, soleIntervalIterator, unmodifiableCurrentIterators

Fields inherited from class it.unimi.di.big.mg4j.search.AbstractDocumentIterator

curr, weight

Fields inherited from interface it.unimi.di.big.mg4j.search.DocumentIterator

END_OF_LIST

Fields inherited from interface it.unimi.di.big.mg4j.index.IndexIterator

END_OF_POSITIONS

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	MultiTermIndexIterator(int defaultFrequency, IndexIterator... indexIterator)	Creates a new document iterator that merges the given array of iterators.

Method Summary

Modifier and Type	Method	Description
<T> T	accept(DocumentIteratorVisitor<T> visitor)	Accepts a visitor.
<T> T	acceptDeep(DocumentIteratorVisitor<T> visitor)
<T> T	acceptDeepOnTruePaths(DocumentIteratorVisitor<T> visitor)
<T> T	acceptOnTruePaths(DocumentIteratorVisitor<T> visitor)	Invokes AbstractUnionDocumentIterator.acceptOnTruePaths(DocumentIteratorVisitor) only on component iterators positioned on the current document.
int	count()	The count is the sum of counts of those component iterators positioned on the current document.
long	frequency()	The frequency is either the default frequency set at construction time, or the maximum frequency of the component iterators.
int	front(IndexIterator[] indexIterator)	Fills the given array with the index iterators composing the current front.
static IndexIterator	getInstance(int defaultFrequency, IndexIterator... indexIterator)	Returns an index iterator that merges the given array of iterators.
static IndexIterator	getInstance(int defaultFrequency, Index index, IndexIterator... indexIterator)	Returns an index iterator that merges the given array of iterators.
static IndexIterator	getInstance(IndexIterator... indexIterator)	Returns an index iterator that merges the given array of iterators.
static IndexIterator	getInstance(Index index, IndexIterator... indexIterator)	Returns an index iterator that merges the given array of iterators.
protected IntervalIterator	getIntervalIterator(Index index, int n, boolean allIndexIterators, Object unused)	Creates an interval iterator suitable for this AbstractIntervalDocumentIterator.
int	id()	Returns the id of this index iterator.
IndexIterator	id(int id)	Sets the id of this index iterator.
Index	index()	Returns the index over which this iterator is built.
IntervalIterator	intervalIterator()	Returns the interval iterator of this document iterator for single-index queries.
IntervalIterator	intervalIterator(Index index)	Returns the interval iterator of this document iterator for the given index.
long	nextDocument()	Returns the next document provided by this document iterator, or DocumentIterator.END_OF_LIST if no more documents are available.
int	nextPosition()	Returns the next position at which the term appears in the current document.
Payload	payload()	Returns the payload, if any, associated with the current document.
long	skipTo(long n)	Skips all documents smaller than n.
String	term()	Returns the term whose inverted list is returned by this index iterator.
IndexIterator	term(CharSequence term)	Sets the term whose inverted list is returned by this index iterator.
long	termNumber()	Returns the number of the term whose inverted list is returned by this index iterator.
IndexIterator	weight(double weight)	Sets the weight of this index iterator.

Methods inherited from class it.unimi.di.big.mg4j.search.AbstractUnionDocumentIterator

computeFront, intervalIterators

Methods inherited from class it.unimi.di.big.mg4j.search.AbstractCompositeDocumentIterator

dispose, toString

Methods inherited from class it.unimi.di.big.mg4j.search.AbstractIntervalDocumentIterator

allIndexIterators, indices, indices

Methods inherited from class it.unimi.di.big.mg4j.search.AbstractDocumentIterator

document, ensureOnADocument, mayHaveNext, weight

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface it.unimi.di.big.mg4j.search.DocumentIterator

dispose, document, indices, intervalIterators, mayHaveNext, weight

Field Detail

term

protected String term

The term of this iterator.

id

protected int id

The id of this iterator.

Constructor Detail

MultiTermIndexIterator

protected MultiTermIndexIterator(int defaultFrequency,
                                 IndexIterator... indexIterator)

Creates a new document iterator that merges the given array of iterators.

Parameters:

defaultFrequency - the default term frequency (or Integer.MIN_VALUE for the max).

indexIterator - the iterators to be joined.

Method Detail

getInstance

public static IndexIterator getInstance(IndexIterator... indexIterator)

Returns an index iterator that merges the given array of iterators. This method requires that at least one iterator is provided. The frequency is computed as a max, and index() will return the result of the same method on the first iterator.

Parameters:

indexIterator - the iterators to be joined (at least one).

Returns:

a merged index iterator.

Throws:

IllegalArgumentException - if no iterators were provided.

getInstance

public static IndexIterator getInstance(Index index,
                                        IndexIterator... indexIterator)

Returns an index iterator that merges the given array of iterators.

Note that the special case of the empty and of the singleton arrays are handled efficiently. The frequency is computed as a max, and index() will return index.

Parameters:

index - the index that wil be returned by index().

indexIterator - the iterators to be joined.

Returns:

a merged index iterator.

getInstance

public static IndexIterator getInstance(int defaultFrequency,
                                        IndexIterator... indexIterator)

Returns an index iterator that merges the given array of iterators. This method requires that at least one iterator is provided.

Parameters:

defaultFrequency - the default term frequency (or Integer.MIN_VALUE for the max).

indexIterator - the iterators to be joined (at least one).

Returns:

a merged index iterator.

Throws:

IllegalArgumentException - if no iterators were provided, or they run on different indices.

getInstance

public static IndexIterator getInstance(int defaultFrequency,
                                        Index index,
                                        IndexIterator... indexIterator)

Returns an index iterator that merges the given array of iterators.

Note that the special case of the empty and of the singleton arrays are handled efficiently.

Parameters:

defaultFrequency - the default term frequency (or Integer.MIN_VALUE for the max).

index - the index that wil be returned by index().

indexIterator - the iterators to be joined.

Returns:

a merged index iterator.

Throws:

IllegalArgumentException - if there is some iterator on an index different from index.

getIntervalIterator

protected IntervalIterator getIntervalIterator(Index index,
                                               int n,
                                               boolean allIndexIterators,
                                               Object unused)

Description copied from class: AbstractIntervalDocumentIterator

Creates an interval iterator suitable for this AbstractIntervalDocumentIterator.

Specified by:

getIntervalIterator in class AbstractIntervalDocumentIterator

Parameters:

index - the reference index for the iterator, or null.

n - the number of underlying or component iterators.

allIndexIterators - whether all underlying or component iterators are index iterators.

unused - an optional argument.

Returns:

an interval iterator suitable for this AbstractIntervalDocumentIterator.

skipTo

public long skipTo(long n)
            throws IOException

Description copied from interface: DocumentIterator

Skips all documents smaller than n.

Define the current document k associated with this document iterator as follows:

• -1, if DocumentIterator.nextDocument() and this method have never been called;
• DocumentIterator.END_OF_LIST, if a call to this method or to DocumentIterator.nextDocument() returned DocumentIterator.END_OF_LIST;
• the last value returned by a call to DocumentIterator.nextDocument() or this method, otherwise.

If k is larger than or equal to n, then this method does nothing and returns k. Otherwise, a call to this method is equivalent to

 while( ( k = nextDocument() ) < n );
 return k;
 

Thus, when a result k ≠ DocumentIterator.END_OF_LIST is returned, the state of this iterator will be exactly the same as after a call to DocumentIterator.nextDocument() that returned k. In particular, the first document larger than or equal to n (when returned by this method) will not be returned by the next call to DocumentIterator.nextDocument().

Specified by:

skipTo in interface DocumentIterator

Overrides:

skipTo in class AbstractUnionDocumentIterator

Parameters:

n - a document pointer.

Returns:

a document pointer larger than or equal to n if available, DocumentIterator.END_OF_LIST otherwise.

Throws:

IOException

nextDocument

public long nextDocument()
                  throws IOException

Description copied from interface: DocumentIterator

Returns the next document provided by this document iterator, or DocumentIterator.END_OF_LIST if no more documents are available.

Specified by:

nextDocument in interface DocumentIterator

Overrides:

nextDocument in class AbstractUnionDocumentIterator

Returns:

the next document, or DocumentIterator.END_OF_LIST if no more documents are available.

Throws:

IOException

count

public int count()
          throws IOException

The count is the sum of counts of those component iterators positioned on the current document.

Specified by:

count in interface IndexIterator

Returns:

the sum of counts.

Throws:

IOException

front

public int front(IndexIterator[] indexIterator)

Fills the given array with the index iterators composing the current front.

This method is essentially a safe exposure of the front of the queue merging the component index iterators. After a call to nextDocument(), you can use this method to know which terms actually appear in the current document. You can use the public field AbstractCompositeDocumentIterator.n to size the argument array appropriately.

Parameters:

indexIterator - an array, at least large as the number of component index iterators, that will be partially filled with the index iterators corresponding to terms appearing in the current document.

Returns:

the number of iterators written into indexIterator.

frequency

public long frequency()
               throws IOException

The frequency is either the default frequency set at construction time, or the maximum frequency of the component iterators.

Specified by:

frequency in interface IndexIterator

Returns:

the frequency.

Throws:

IOException

term

public IndexIterator term(CharSequence term)

Description copied from interface: IndexIterator

Sets the term whose inverted list is returned by this index iterator.

Usually, the term is automatically set by Index.documents(CharSequence) or by IndexReader.documents(CharSequence), but you can use this method to ensure that IndexIterator.term() doesn't throw an exception.

Specified by:

term in interface IndexIterator

Parameters:

term - a character sequence (that will be defensively copied) that will be assumed to be the term whose inverted list is returned by this index iterator.

Returns:

this index iterator.

term

public String term()

Description copied from interface: IndexIterator

Returns the term whose inverted list is returned by this index iterator.

Usually, the term is automatically set by IndexReader.documents(CharSequence) or IndexReader.documents(long), but you can supply your own term with IndexIterator.term(CharSequence).

Specified by:

term in interface IndexIterator

Returns:

the term over which this iterator is built, as a compact mutable string.

See Also:

IndexIterator.termNumber()

termNumber

public long termNumber()

Description copied from interface: IndexIterator

Returns the number of the term whose inverted list is returned by this index iterator.

Usually, the term number is automatically set by IndexReader.documents(CharSequence) or IndexReader.documents(long).

Specified by:

termNumber in interface IndexIterator

Returns:

the number of the term over which this iterator is built.

See Also:

IndexIterator.term()

id

public IndexIterator id(int id)

Description copied from interface: IndexIterator

Sets the id of this index iterator.

The id is an integer associated with each index iterator. It has no specific semantics, and can be used differently in different contexts. A typical usage pattern, for instance, is using it to assign a unique number to the index iterators contained in a composite document iterator (say, numbering consecutively the leaves of the composite).

Specified by:

id in interface IndexIterator

Parameters:

id - the new id for this index iterator.

Returns:

this index iterator.

id

public int id()

Description copied from interface: IndexIterator

Returns the id of this index iterator.

Specified by:

id in interface IndexIterator

Returns:

the id of this index iterator.

See Also:

IndexIterator.id(int)

index

public Index index()

Description copied from interface: IndexIterator

Returns the index over which this iterator is built.

Specified by:

index in interface IndexIterator

Returns:

the index over which this iterator is built.

payload

public Payload payload()

Description copied from interface: IndexIterator

Returns the payload, if any, associated with the current document.

Specified by:

payload in interface IndexIterator

Returns:

the payload associated with the current document.

weight

public IndexIterator weight(double weight)

Description copied from interface: DocumentIterator

Sets the weight of this index iterator.

Specified by:

weight in interface DocumentIterator

Specified by:

weight in interface IndexIterator

Overrides:

weight in class AbstractDocumentIterator

Parameters:

weight - the weight of this index iterator.

Returns:

this document iterator.

See Also:

DocumentIterator.weight(double)

accept

public <T> T accept(DocumentIteratorVisitor<T> visitor)
             throws IOException

Description copied from interface: DocumentIterator

Accepts a visitor.

A document iterator is usually structured as composite, with operators as internal nodes and IndexIterators as leaves. This method implements the visitor pattern.

Specified by:

accept in interface DocumentIterator

Overrides:

accept in class AbstractCompositeDocumentIterator

Parameters:

visitor - the visitor.

Returns:

an object resulting from the visit, or null if the visit was interrupted.

Throws:

IOException

acceptOnTruePaths

public <T> T acceptOnTruePaths(DocumentIteratorVisitor<T> visitor)
                        throws IOException

Description copied from class: AbstractUnionDocumentIterator

Invokes AbstractUnionDocumentIterator.acceptOnTruePaths(DocumentIteratorVisitor) only on component iterators positioned on the current document.

Specified by:

acceptOnTruePaths in interface DocumentIterator

Overrides:

acceptOnTruePaths in class AbstractUnionDocumentIterator

Parameters:

visitor - a visitor.

Returns:

true if the visit should continue.

Throws:

IOException

See Also:

DocumentIterator.accept(DocumentIteratorVisitor), CounterCollectionVisitor

acceptDeep

public <T> T acceptDeep(DocumentIteratorVisitor<T> visitor)
                 throws IOException

Throws:

IOException

acceptDeepOnTruePaths

public <T> T acceptDeepOnTruePaths(DocumentIteratorVisitor<T> visitor)
                            throws IOException

Throws:

IOException

nextPosition

public int nextPosition()
                 throws IOException

Description copied from interface: IndexIterator

Returns the next position at which the term appears in the current document.

Specified by:

nextPosition in interface IndexIterator

Returns:

the next position of the current document in which the current term appears, or IndexIterator.END_OF_POSITIONS if there are no more positions.

Throws:

IOException

intervalIterator

public IntervalIterator intervalIterator()

Description copied from interface: DocumentIterator

Returns the interval iterator of this document iterator for single-index queries.

This is a commodity method that can be used only for queries built over a single index.

Specified by:

intervalIterator in interface DocumentIterator

Returns:

an interval iterator.

See Also:

DocumentIterator.intervalIterator(Index)

intervalIterator

public IntervalIterator intervalIterator(Index index)

Description copied from interface: DocumentIterator

Returns the interval iterator of this document iterator for the given index.

After a call to DocumentIterator.nextDocument(), this iterator can be used to retrieve the intervals in the current document (the one returned by DocumentIterator.nextDocument()) for the index index.

Note that if all indices have positions, it is guaranteed that at least one index will return an interval. However, for disjunctive queries it cannot be guaranteed that all indices will return an interval.

Indices without positions always return IntervalIterators.TRUE. Thus, in presence of indices without positions it is possible that no intervals at all are available.

Specified by:

intervalIterator in interface DocumentIterator

Parameters:

index - an index (must be one over which the query was built).

Returns:

an interval iterator over the current document in index.