org.neo4j.com.storecopy

Class TransactionCommittingResponseUnpacker

java.lang.Object

org.neo4j.kernel.lifecycle.LifecycleAdapter

org.neo4j.com.storecopy.TransactionCommittingResponseUnpacker

All Implemented Interfaces:

ResponseUnpacker, org.neo4j.kernel.lifecycle.Lifecycle

public class TransactionCommittingResponseUnpacker
extends org.neo4j.kernel.lifecycle.LifecycleAdapter
implements ResponseUnpacker

Receives and unpacks responses. Transaction obligations are handled by TransactionObligationFulfiller and transaction streams are committed to the store, in batches.

It is assumed that any response carrying transaction data comes from the one and same thread.

SAFE ZONE EXPLAINED

PROBLEM A slave can read inconsistent or corrupted data (mixed state records) because of reuse of property ids. This happens when a record that has been read gets reused and then read again or possibly reused in middle of reading a property chain or dynamic record chain. This is guarded for in single instance with the delaying of id reuse. This does not cover the Slave case because the transactions that SET, REMOVE and REUSE the record are applied in batch and thus a slave transaction can see states from all of the transactions that touches the reused record during its lifetime.

SOLUTION Master and Slave are configured with the same safeZone time. Let S = safeZone time (more about safeZone time further down)

-> Master promise to hold all deleted ids in quarantine before reusing them, (S duration). He thereby creates a safe zone of transactions that among themselves are guaranteed to be free of id reuse contamination. -> Slave promise to not let any transactions cross the safe zone boundary. Meaning all transactions that falls out of the safe zone, as updates gets applied, will need to be terminated, with a hint that they can simply be restarted

Safe zone is a time frame in Masters domain. All transactions that started and finished within this time frame are guarantied to not have read any mixed state records.

Example of a transaction running on slave that starts reading a dynamic property, then a batch is pulled from master that deletes the property and and reuses the record in the chain, making the transaction read inconsistent data.

TX starts reading tx here

 v
 |aaaa|->|aaaa|->|aaaa|->|aaaa|
 1       2       3       4
 

"a" string is deleted and replaced with "bbbbbbbbbbbbbbbb"

 tx here
 v
 |bbbb|->|bbbb|->|bbbb|->|bbbb|
 1       2       3       4
 

TX continues reading and does not know anything is wrong, returning the inconsistent string "aaaaaaaabbbbbbbb".

 tx here
 v
 |bbbb|->|bbbb|->|bbbb|->|bbbb|
 1       2       3       4
 

Example of how the safe zone window moves while appying a batch

 x---------------------------------------------------------------------------------->| TIME
 |MASTER STATE
 |---------------------------------------------------------------------------------->|
 |                                                          Batch to apply to slave
 |                                  safeZone with size S  |<------------------------>|
 |                                                  |
 |                                                  v     A
 |SLAVE STATE 1 (before applying batch)         |<---S--->|
 |----------------------------------------------+-------->|
 |                                                        |
 |                                                        |
 |                                                        |      B
 |SLAVE STATE 2 (mid apply)                            |<-+-S--->|
 |-----------------------------------------------------+--+----->|
 |                                                        |      |
 |                                                        |      |
 |                                                        |      |  C
 |SLAVE STATE 3 (mid apply / after apply)                 |<---S-+->|
 |--------------------------------------------------------+------+->|
 |                                                        |      |  |
 |                                                        |      |  |
 |                                                        |      |  |                D
 |SLAVE STATE 4 (after apply)                             |      |  |      |<---S--->|
 |--------------------------------------------------------+------+--+------+-------->|
 

What we see in this diagram is a slave pulling updates from the master. While doing so, the safe zone window |<---S--->| is pushed forward. NOTE that we do not see any explicit transaction running on slave. Only the times (A, B, C, D) that we discuss.

slaveTx start on slave when slave is in SLAVE STATE 1 - Latest applied transaction on slave has timestamp A and safe zone is A-S. - slaveTx.startTime = A

Scenario 1 - slaveTx finish when slave is in SLAVE STATE 2 Latest applied transaction in store has timestamp B and safe zone is B-S. slaveTx did not cross the safe zone boundary as slaveTx.startTime = A > B-S We can safely assume that slaveTx did not read any mixed state records.

Scenario 2 - slaveTx has not yet finished in SLAVE STATE 3 Latest applied transaction in store has timestamp C and safe zone is C-S. We are just about to apply the next part of the batch and push the safe zone window forward. This will make slaveTx.startTime = A < C-S. This means Tx is now in risk of reading mixed state records. We will terminate slaveTx and let the user try again.

NOTE ABOUT TX_COMMIT_TIMESTAMP commitTimestamp is used by MetaDataStore to keep track of the commit timestamp of the last committed transaction. When starting up a db we can not always know what the the latest commit timestamp is but slave need it to know when a transaction needs to be terminated during batch application. The latest commit timestamp is an important part of "safeZone" that is explained in TransactionCommittingResponseUnpacker.

Here are the different scenarios, what timestamp that is used and what it means for execution.

Empty store
TIMESTAMP: TransactionIdStore.BASE_TX_COMMIT_TIMESTAMP
==> FINE. NO KILL because no previous state can have been observed anyway

Upgraded store w/ tx logs
TIMESTAMP CARRIED OVER FROM LOG
==> FINE

Upgraded store w/o tx logs
TIMESTAMP TransactionIdStore.UNKNOWN_TX_COMMIT_TIMESTAMP (1)
==> SLAVE TRANSACTIONS WILL TERMINATE WHEN FIRST PULL UPDATES HAPPENS

Store on 2.3.prev, w/ tx logs (no upgrade)
TIMESTAMP CARRIED OVER FROM LOG
==> FINE

Store on 2.3.prev w/o tx logs (no upgrade)
TIMESTAMP TransactionIdStore.UNKNOWN_TX_COMMIT_TIMESTAMP (1)
==> SLAVE TRANSACTIONS WILL TERMINATE WHEN FIRST PULL UPDATES HAPPENS

Store already on 2.3.next, w/ or w/o tx logs
TIMESTAMP CORRECT
==> FINE

See Also:

TransactionBatchCommitter

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	TransactionCommittingResponseUnpacker.Dependencies Dependencies that this TransactionCommittingResponseUnpacker has.

Nested classes/interfaces inherited from interface org.neo4j.com.storecopy.ResponseUnpacker

ResponseUnpacker.TxHandler

Nested classes/interfaces inherited from interface org.neo4j.kernel.lifecycle.Lifecycle

org.neo4j.kernel.lifecycle.Lifecycle.Delegate

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_BATCH_SIZE

Fields inherited from interface org.neo4j.com.storecopy.ResponseUnpacker

NO_OP_RESPONSE_UNPACKER

Constructor Summary

Constructors

Constructor and Description
TransactionCommittingResponseUnpacker(org.neo4j.graphdb.DependencyResolver dependencies, int maxBatchSize, long idReuseSafeZoneTime)
TransactionCommittingResponseUnpacker(TransactionCommittingResponseUnpacker.Dependencies dependencies, int maxBatchSize, long idReuseSafeZoneTime)

Method Summary

Modifier and Type	Method and Description
void	start()
void	stop()
void	unpackResponse(Response<?> response, ResponseUnpacker.TxHandler txHandler)

Methods inherited from class org.neo4j.kernel.lifecycle.LifecycleAdapter

init, shutdown

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_BATCH_SIZE

public static final int DEFAULT_BATCH_SIZE

See Also:

Constant Field Values

Constructor Detail

TransactionCommittingResponseUnpacker

public TransactionCommittingResponseUnpacker(org.neo4j.graphdb.DependencyResolver dependencies,
                                             int maxBatchSize,
                                             long idReuseSafeZoneTime)

TransactionCommittingResponseUnpacker

public TransactionCommittingResponseUnpacker(TransactionCommittingResponseUnpacker.Dependencies dependencies,
                                             int maxBatchSize,
                                             long idReuseSafeZoneTime)

Method Detail

unpackResponse

public void unpackResponse(Response<?> response,
                           ResponseUnpacker.TxHandler txHandler)
                    throws Exception

Specified by:

unpackResponse in interface ResponseUnpacker

txHandler - for getting an insight into which transactions gets applied.

Throws:

Exception

start

public void start()

Specified by:

start in interface org.neo4j.kernel.lifecycle.Lifecycle

Overrides:

start in class org.neo4j.kernel.lifecycle.LifecycleAdapter

stop

public void stop()

Specified by:

stop in interface org.neo4j.kernel.lifecycle.Lifecycle

Overrides:

stop in class org.neo4j.kernel.lifecycle.LifecycleAdapter