Package edu.berkeley.nlp.lm.io
Class KneserNeyFileWritingLmReaderCallback<W>
- java.lang.Object
-
- edu.berkeley.nlp.lm.io.KneserNeyFileWritingLmReaderCallback<W>
-
- Type Parameters:
W
-
- All Implemented Interfaces:
ArpaLmReaderCallback<ProbBackoffPair>
,LmReaderCallback<ProbBackoffPair>
,NgramOrderedLmReaderCallback<ProbBackoffPair>
public class KneserNeyFileWritingLmReaderCallback<W> extends java.lang.Object implements ArpaLmReaderCallback<ProbBackoffPair>
Class for producing a Kneser-Ney language model in ARPA format from raw text.- Author:
- adampauls
-
-
Constructor Summary
Constructors Constructor Description KneserNeyFileWritingLmReaderCallback(java.io.File outputFile, WordIndexer<W> wordIndexer)
KneserNeyFileWritingLmReaderCallback(java.io.PrintWriter out, WordIndexer<W> wordIndexer)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description void
call(int[] ngram, int startPos, int endPos, ProbBackoffPair value, java.lang.String words)
Called for each n-gramvoid
cleanup()
Called once all reading is done.void
handleNgramOrderFinished(int order)
Called when all n-grams of a given order are finishedvoid
handleNgramOrderStarted(int order)
Called when n-grams of a given order are startedvoid
initWithLengths(java.util.List<java.lang.Long> numNGrams)
Called initially with a list of how many n-grams will appear for each order.
-
-
-
Constructor Detail
-
KneserNeyFileWritingLmReaderCallback
public KneserNeyFileWritingLmReaderCallback(java.io.File outputFile, WordIndexer<W> wordIndexer)
-
KneserNeyFileWritingLmReaderCallback
public KneserNeyFileWritingLmReaderCallback(java.io.PrintWriter out, WordIndexer<W> wordIndexer)
-
-
Method Detail
-
handleNgramOrderFinished
public void handleNgramOrderFinished(int order)
Description copied from interface:NgramOrderedLmReaderCallback
Called when all n-grams of a given order are finished- Specified by:
handleNgramOrderFinished
in interfaceNgramOrderedLmReaderCallback<W>
-
handleNgramOrderStarted
public void handleNgramOrderStarted(int order)
Description copied from interface:NgramOrderedLmReaderCallback
Called when n-grams of a given order are started- Specified by:
handleNgramOrderStarted
in interfaceNgramOrderedLmReaderCallback<W>
-
call
public void call(int[] ngram, int startPos, int endPos, ProbBackoffPair value, java.lang.String words)
Description copied from interface:LmReaderCallback
Called for each n-gram- Specified by:
call
in interfaceLmReaderCallback<W>
- Parameters:
ngram
- The integer representation of the words as given by the provided WordIndexervalue
- The value of the n-gramwords
- The string representation of the n-gram (space separated)
-
cleanup
public void cleanup()
Description copied from interface:LmReaderCallback
Called once all reading is done.- Specified by:
cleanup
in interfaceLmReaderCallback<W>
-
initWithLengths
public void initWithLengths(java.util.List<java.lang.Long> numNGrams)
Description copied from interface:ArpaLmReaderCallback
Called initially with a list of how many n-grams will appear for each order.- Specified by:
initWithLengths
in interfaceArpaLmReaderCallback<W>
- Parameters:
numNGrams
- maps n-gram orders to number of n-grams (i.e. numNGrams.get(0) is the number of unigrams)
-
-