BulkSqlUtil::BulkInsertOperation Class Reference

base class for bulk DML insert operations More...

Inheritance diagram for BulkSqlUtil::BulkInsertOperation:

Public Member Methods
	constructor (SqlUtil::AbstractTable target, *hash opts)
	creates the object from the supplied arguments More...
	constructor (SqlUtil::Table target, *hash opts)
	creates the object from the supplied arguments More...
	setRowCode (*code rowc)
	sets a closure or call reference that will be called when data has been sent to the database and all output data is available; must accept a hash argument that represents the data written to the database including any output arguments. This code will be reset, once the transaction is commited. More...
Public Member Methods inherited from BulkSqlUtil::AbstractBulkOperation
nothing	commit ()
	flushes any queued data and commits the transaction
	constructor (string name, SqlUtil::AbstractTable target, *hash opts)
	creates the object from the supplied arguments More...
	constructor (string name, SqlUtil::Table target, *hash opts)
	creates the object from the supplied arguments More...
	destructor ()
	throws an exception if there is data pending in the internal row data cache; make sure to call flush() or discard() before destroying the object More...
	discard ()
	discards any buffered batched data; this method should be called before destroying the object if an error occurs More...
	flush ()
	flushes any remaining batched data to the database; this method should always be called before committing the transaction or destroying the object More...
Qore::SQL::AbstractDatasource	getDatasource ()
	returns the AbstractDatasource object associated with this object
int	getRowCount ()
	returns the affected row count
SqlUtil::AbstractTable	getTable ()
	returns the underlying SqlUtil::AbstractTable object
string	getTableName ()
	returns the table name
	queueData (hash data)
	queues row data in the block buffer; the block buffer is flushed to the DB if the buffer size reaches the limit defined by the block_size option; does not commit the transaction More...
	queueData (list l)
	queues row data in the block buffer; the block buffer is flushed to the DB if the buffer size reaches the limit defined by the block_size option; does not commit the transaction More...
nothing	rollback ()
	discards any queued data and rolls back the transaction
int	size ()
	returns the current size of the cache as a number of rows More...

Private Member Methods
	flushImpl ()
	inserts internally-queued queued data in the database with bulk DML operations More...
	init (*hash opts)
	common constructor initialization
	setupInitialRow (hash row)
	sets up support for "returning" insert options for any possible rowcode member
	setupInitialRowColumns (hash row)
	sets up the block buffer given the initial template hash of lists for inserting
Private Member Methods inherited from BulkSqlUtil::AbstractBulkOperation
	flushIntern ()
	flushes queued data to the database
	init (*hash opts)
	common constructor initialization
	setupInitialRow (hash row)
	sets up the block buffer given the initial template row for inserting
	setupInitialRowColumns (hash row)
	sets up the block buffer given the initial template hash of lists for inserting

Private Attributes
*code	rowcode
	per-row Closures or Call References for inserts
hash	static_ret_expr
	hash of "returning" arguments
AbstractSQLStatement	stmt
	statement for DML
Private Attributes inherited from BulkSqlUtil::AbstractBulkOperation
softint	block_size
	bulk operation block size
hash	cval
	"constant" row values; must be equal in all calls to queueData
list	cval_keys
	"constant" row value keys
hash	hbuf
	buffer for bulk operations
*code	info_log
	an optional info logging callback; must accept a sprintf()-style format specifier and optional arguments
string	opname
	operation name
list	ret_args = ()
	list of "returning" columns
int	row_count = 0
	row count
SqlUtil::AbstractTable	table
	the target table object

Additional Inherited Members
Public Attributes inherited from BulkSqlUtil::AbstractBulkOperation
const	OptionDefaults = ...
	default option values
const	OptionKeys = ...
	option keys for this object

Detailed Description

base class for bulk DML insert operations

This class assists with bulk inserts into a target table.

Submitting Data

To use this class, queue data in the form of a hash (a single row or a set of rows) or a list of rows by calling the queueData() method.

The queueData() method queues data to be written to the database; the queue is flush()ed automatically when block_size rows have been queued.

Retrieving Data From Inserts

It is possible to use SQL Insert Operator Functions in the hashes submitted with queueData(); in this case the BulkInsertOperation class assumes that every row has the same operations as in the first row. Output data can then be processed by using the rowcode option in the constructor() or by calling setRowCode().

In case SQL Insert Operator Functions are used and a rowcode option is set, then the SQL DML query for inserts is creating using the "returning" insert option, therefore the DBI driver in this case must support this option as well.

Flushing and Discarding Data

Each call to flush() (whether implicit or explicit) will cause a single call to be made to the dataserver; all queued rows are sent in a single bulk DML call, which allows for efficient processing of large amounts of data.

A call to flush() must be made before committing the transaction to ensure that any remaining rows in the internal queue have been written to the database. Because the destructor() will throw an exception if any data is left in the internal queue when the object is destroyed, a call to discard() must be made prior to the destruction of the object in case of errors.

# single commit and rollback
on_success ds.commit();
on_error ds.rollback();
{
    BulkInsertOperation op1(table1);
    BulkInsertOperation op2(table2);
 
    # each operation needs to be flushed or discarded individually
    on_success {
        op1.flush();
        op2.flush();
    }
    on_error {
        op1.discard();
        op2.discard();
    }
 
    # data is queued and flushed automatically when the buffer is full
    map op1.queueData($1), data1.iterator();
    map op2.queueData($1), data2.iterator();
}

Note

Each bulk DML object must be manually flush()ed before committing or manually discard()ed before rolling back to ensure that all data is managed properly in the same transaction and to ensure that no exception is thrown in the destructor(). See the example above for more information.

Member Function Documentation

constructor() [1/2]

BulkSqlUtil::BulkInsertOperation::constructor	(	SqlUtil::AbstractTable	target,
		*hash	opts
	)

creates the object from the supplied arguments

Parameters

target

the target table object

opts

an optional hash of options for the object as follows: "info_log": an optional info logging callback; must accept a string format specifier and sprintf()-style arguments "block_size": the number of rows executed at once (default: 1000) "rowcode": a per-row Closures or Call References for batch inserts; this must take a single hash argument and will be called for every row after a bulk insert; the hash argument representing the row inserted will also contain any output values if applicable (for example if SQL Insert Operator Functions are used in the row hashes submitted to queueData())

See also

setRowCode()

constructor() [2/2]

BulkSqlUtil::BulkInsertOperation::constructor	(	SqlUtil::Table	target,
		*hash	opts
	)

creates the object from the supplied arguments

Parameters

target

the target table object

opts

an optional hash of options for the object as follows: "info_log": an optional info logging callback; must accept a string format specifier and sprintf()-style arguments "block_size": the number of rows executed at once (default: 1000) "rowcode": a per-row Closures or Call References for batch inserts; this must take a single hash argument and will be called for every row after a bulk insert; the hash argument representing the row inserted will also contain any output values if applicable (for example if SQL Insert Operator Functions are used in the row hashes submitted to queueData())

See also

setRowCode()

flushImpl()

BulkSqlUtil::BulkInsertOperation::flushImpl ( )

privatevirtual

inserts internally-queued queued data in the database with bulk DML operations

This method sets up the SQL DML query used for inserts when row is queued. Output values are only retrieved if SQL Insert Operator Functions are used and a rowcode closure or call reference has been set beforehand in the constructor() or by calling setRowCode() and the underlying DBI driver supports the "returning" insert option.

Implements BulkSqlUtil::AbstractBulkOperation.

setRowCode()

BulkSqlUtil::BulkInsertOperation::setRowCode

(

*code

rowc

)

sets a closure or call reference that will be called when data has been sent to the database and all output data is available; must accept a hash argument that represents the data written to the database including any output arguments. This code will be reset, once the transaction is commited.

Example:

# single commit and rollback
on_success ds.commit();
on_error ds.rollback();
code rowcode = sub (hash row) {
    # process row data
};
inserter.setRowCode(rowcode);
{
    # each operation needs to be flushed or discarded individually
    on_success inserter.flush();
    on_error inserter.discard();
 
    # data is queued and flushed automatically when the buffer is full
    map inserter.queueData($1), data.iterator();
}

Parameters

rowc	a closure or call reference that will be called when data has been sent to the database and all output data is available; must accept a hash argument that represents the data written to the database including any output arguments

Note

• the per-row closure or call reference can also be set by using the "rowcode" option in the constructor()
• if this method is not called before the first row is queued then output values will not be retrieved; the initial query is built when the template row is queued and output values are only retrieved if a rowcode closure or call reference is set beforehand