Package org.wikidata.wdtk.client

Class DumpProcessingOutputAction

java.lang.Object

org.wikidata.wdtk.client.DumpProcessingOutputAction

All Implemented Interfaces:

DumpProcessingAction, org.wikidata.wdtk.datamodel.interfaces.EntityDocumentDumpProcessor, org.wikidata.wdtk.datamodel.interfaces.EntityDocumentProcessor

Direct Known Subclasses:

JsonSerializationAction, RdfSerializationAction

public abstract class DumpProcessingOutputAction
extends java.lang.Object
implements DumpProcessingAction

Abstract implementation of DumpProcessingAction that represents actions which generate outputs (in the form of files). It supports options OPTION_DESTINATION, OPTION_COMPRESSION, and OPTION_USE_STDOUT. Moreover, it provides some static helper functions for opening files for writing.

Author:

Michael Günther, Markus Kroetzsch

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
protected static interface	DumpProcessingOutputAction.FinishableRunnable	Simple interface for a Runnable that can be stopped gracefully by calling a method DumpProcessingOutputAction.FinishableRunnable.finish().

Field Summary

Fields

Modifier and Type	Field	Description
static java.lang.String	COMPRESS_BZ2
static java.lang.String	COMPRESS_GZIP
static java.lang.String	COMPRESS_NONE
protected java.lang.String	compressionType	String to indicate the output compression to be used, if any.
protected java.lang.String	dateStamp	Date stamp of the dump to be processed.
protected java.lang.String	name	The name of the action.
static java.lang.String	OPTION_COMPRESSION	Name of the option used to define that output be compressed in a particular way.
static java.lang.String	OPTION_DESTINATION	Name of the option to set the destination file name (may include path) for outputs generated by this action.
static java.lang.String	OPTION_USE_STDOUT	Name of the option to redirect output to stdout.
protected java.lang.String	outputDestination	For operations that generate output, this is the name of the output file, or null to use the default file name for the given options.
protected java.util.Set<java.io.Closeable>	outputStreams	Output streams that were created by this class.
protected java.lang.String	project	String name of the site that the processed dump file comes from.
protected org.wikidata.wdtk.datamodel.interfaces.Sites	sites	The Sites object if provided.
protected boolean	useStdOut	Specifies if output should be redirected to stdout rather than being written to a file.

Constructor Summary

Constructors

Constructor	Description
DumpProcessingOutputAction()

Method Summary

Modifier and Type	Method	Description
void	close()
java.lang.String	getActionName()	Returns the name of the action.
protected java.io.OutputStream	getAsynchronousOutputStream(java.io.OutputStream outputStream)	Creates a separate thread for writing into the given output stream and returns a pipe output stream that can be used to pass data to this thread.
java.lang.String	getOutputFilename()
protected java.io.OutputStream	getOutputStream(boolean useStdOut, java.lang.String filePath, java.lang.String compressionType)	Creates an compressing OutputStream.
java.lang.String	insertDumpInformation(java.lang.String pattern)
void	setActionName(java.lang.String name)	Sets the name of the action.
void	setDumpInformation(java.lang.String project, java.lang.String dateStamp)	Provides the action with general information about the dump that is to be processed.
boolean	setOption(java.lang.String option, java.lang.String value)	Sets the options of the specified name to the given value.
void	setSites(org.wikidata.wdtk.datamodel.interfaces.Sites sites)	Sets the sites information to the given value.
boolean	useStdOut()	Returns true if this action will write results (not log messages) to stdout.

Methods inherited from interface org.wikidata.wdtk.client.DumpProcessingAction

getDefaultActionName, getReport, isReady, needsSites

Methods inherited from interface org.wikidata.wdtk.datamodel.interfaces.EntityDocumentDumpProcessor

open

Methods inherited from interface org.wikidata.wdtk.datamodel.interfaces.EntityDocumentProcessor

processItemDocument, processPropertyDocument

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

OPTION_COMPRESSION

public static final java.lang.String OPTION_COMPRESSION

Name of the option used to define that output be compressed in a particular way. By default, output is not compressed. When using compression, the output file will be extended with a suitable file extension.

See Also:

Constant Field Values

OPTION_DESTINATION

public static final java.lang.String OPTION_DESTINATION

Name of the option to set the destination file name (may include path) for outputs generated by this action. If not given, actions should pick a default output file.

See Also:

Constant Field Values

OPTION_USE_STDOUT

public static final java.lang.String OPTION_USE_STDOUT

Name of the option to redirect output to stdout. This is achieved by setting the value to the string "true". Other values are ignored.

See Also:

Constant Field Values

COMPRESS_BZ2

public static final java.lang.String COMPRESS_BZ2

See Also:

Constant Field Values

COMPRESS_GZIP

public static final java.lang.String COMPRESS_GZIP

See Also:

Constant Field Values

COMPRESS_NONE

public static final java.lang.String COMPRESS_NONE

See Also:

Constant Field Values

outputStreams

protected java.util.Set<java.io.Closeable> outputStreams

Output streams that were created by this class. If close is called, it will close all of them properly.

name

protected java.lang.String name

The name of the action.

sites

protected org.wikidata.wdtk.datamodel.interfaces.Sites sites

The Sites object if provided.

See Also:

DumpProcessingAction.needsSites()

outputDestination

protected java.lang.String outputDestination

For operations that generate output, this is the name of the output file, or null to use the default file name for the given options.

useStdOut

protected boolean useStdOut

Specifies if output should be redirected to stdout rather than being written to a file.

compressionType

protected java.lang.String compressionType

String to indicate the output compression to be used, if any.

dateStamp

protected java.lang.String dateStamp

Date stamp of the dump to be processed.

project

protected java.lang.String project

String name of the site that the processed dump file comes from.

Constructor Detail

DumpProcessingOutputAction

public DumpProcessingOutputAction()

Method Detail

setSites

public void setSites(org.wikidata.wdtk.datamodel.interfaces.Sites sites)

Description copied from interface: DumpProcessingAction

Sets the sites information to the given value. The method DumpProcessingAction.needsSites() is used to find out if this is actually needed.

Specified by:

setSites in interface DumpProcessingAction

Parameters:

sites - the sites information for the data that will be processed

setOption

public boolean setOption(java.lang.String option,
                         java.lang.String value)

Description copied from interface: DumpProcessingAction

Sets the options of the specified name to the given value. Returns true if the option was known, and false otherwise. Implementation should overwrite this function to support additional options.

Specified by:

setOption in interface DumpProcessingAction

Parameters:

option - name of the option to be set

value - the new value of the option

Returns:

true if the option was known (no matter if the given value could actually be used or not)

useStdOut

public boolean useStdOut()

Description copied from interface: DumpProcessingAction

Returns true if this action will write results (not log messages) to stdout. The default implementation returns true if no other output destination has been specified. Subclasses that do not write results to stdout in this case should overwrite this method.

Specified by:

useStdOut in interface DumpProcessingAction

Returns:

true if the action is configured to write results to stdout

setDumpInformation

public void setDumpInformation(java.lang.String project,
                               java.lang.String dateStamp)

Description copied from interface: DumpProcessingAction

Provides the action with general information about the dump that is to be processed. This may be used, e.g., to define file names to use for the output.

Specified by:

setDumpInformation in interface DumpProcessingAction

Parameters:

project - the name of the project that the dump is from

dateStamp - the datestamp (YYYYMMDD) for the dump

setActionName

public void setActionName(java.lang.String name)

Description copied from interface: DumpProcessingAction

Sets the name of the action. If this is not set, a default name will be used.

Specified by:

setActionName in interface DumpProcessingAction

getActionName

public java.lang.String getActionName()

Description copied from interface: DumpProcessingAction

Returns the name of the action.

Specified by:

getActionName in interface DumpProcessingAction

Returns:

name

getOutputFilename

public java.lang.String getOutputFilename()

insertDumpInformation

public java.lang.String insertDumpInformation(java.lang.String pattern)

close

public void close()

Specified by:

close in interface org.wikidata.wdtk.datamodel.interfaces.EntityDocumentDumpProcessor

getOutputStream

protected java.io.OutputStream getOutputStream(boolean useStdOut,
                                               java.lang.String filePath,
                                               java.lang.String compressionType)
                                        throws java.io.IOException

Creates an compressing OutputStream. The result is owned by the caller and should be closed later. Neverhteless, the close() method of this class must also be called, since it may free additional resources created.

Parameters:

useStdOut - if true, System.out is returned and the other parameters are ignored

filePath - the string name of the output file, possibly including path information

compressionType - a string that refers to a type of output compression or the empty string (no compression); a suitable file extension will be added to the output file

Returns:

compressing OutputStream

Throws:

java.io.IOException - if there were problems opening the required streams

getAsynchronousOutputStream

protected java.io.OutputStream getAsynchronousOutputStream(java.io.OutputStream outputStream)
                                                    throws java.io.IOException

Creates a separate thread for writing into the given output stream and returns a pipe output stream that can be used to pass data to this thread.

This code is inspired by http://stackoverflow.com/questions/12532073/gzipoutputstream -that-does-its-compression-in-a-separate-thread

Parameters:

outputStream - the stream to write to in the thread

Returns:

a new stream that data should be written to

Throws:

java.io.IOException - if the pipes could not be created for some reason