Qore Mapper Module Reference
1.6
|
this class is a base class for mapping data; see Mapper Examples for usage examples More...
Inherits AbstractDataProcessor.
Public Member Methods | |
commit () | |
Commits data written to the output data provider if the output data provider supports transaction management. More... | |
constructor (hash< auto > mapv, *hash< auto > opts) | |
builds the object based on a hash providing field mappings, data constraints, and optionally custom mapping logic More... | |
discardOutput () | |
Discards any buffered data in the output data provider in a bulk operation. More... | |
flushOutput () | |
Flushes any remaining data to the output data provider in a bulk operation. More... | |
int | getCount () |
returns the internal record count More... | |
string | getFieldName (string fname) |
returns a descriptive name of the given field if possible, otherwise returns the field name itself | |
*AbstractDataProvider | getInputProvider () |
returns the input provider | |
*hash< string, AbstractDataField > | getInputRecord () |
returns the value of the "input" option | |
hash< auto > | getOptionsFromTemplate (hash< auto > opts, hash< auto > input_rec, *hash< auto > output_rec) |
Returns options from a template hash. | |
MapperOutputRecordIterator | getOutputIterator () |
returns an output record iterator that produces mapped data from the input data provider More... | |
*AbstractDataProvider | getOutputProvider () |
returns the output provider | |
*hash< string, AbstractDataField > | getOutputRecord () |
returns the value of the "output" option | |
auto | getRuntime (string key) |
get current runtime option value for a key More... | |
hash< string, hash< MapperOptionInfo > > | getUserOptions () |
returns mapper options useful for users | |
logOutput (hash< auto > h) | |
calls the output logging closure or call reference (if any) to log the output record | |
list< hash< auto > > | mapAll (hash< auto > recs) |
maps all input records and returns the mapped data as a list of output records More... | |
list< hash< auto > > | mapAll (list< auto > recs) |
maps all input records and returns the mapped data as a list of output records More... | |
auto | mapAuto (auto recs) |
maps all input record(s) automatically and returns the mapped data More... | |
*softlist< hash< auto > > | mapAutoInput () |
Runs the input data provider through the mapper and returns the output. More... | |
*hash< auto > | mapBulk (hash< auto > rec, *hash< auto > crec) |
maps a set of records in hash of lists format; returns mapped data in a hash of lists format More... | |
hash< auto > | mapData (hash< auto > rec) |
processes the input record and returns a hash of the mapped values where the keys in the hash returned are the target field names; the order of the fields in the hash returned is the same order as the keys in the map hash. More... | |
hash< auto > | optionKeys () |
returns a list of valid constructor options for this class (can be overridden in subclasses) More... | |
replaceRuntime (*hash< auto > runtime) | |
replaces runtime options More... | |
resetCount () | |
resets the internal record count More... | |
rollback () | |
Rolls back data written to the output data provider. More... | |
runAutonomous () | |
Runs the input and output mappers with data providers on each end autonomously. More... | |
setNullableOutput () | |
sets all fields in the output record as nullable More... | |
setRuntime (hash< auto > runtime) | |
adds runtime options to the current runtime option hash More... | |
setRuntime (string key, auto value) | |
set the runtime option with "key" to value "value" More... | |
hash< string, bool > | validKeys () |
returns a list of valid field keys for this class (can be overridden in subclasses) More... | |
Static Public Member Methods | |
static *hash< auto > | getMapperThreadContext () |
Get mapper thread context. | |
static Type | recordType ("hash<string, AbstractDataField>") |
the input and output record type | |
static list< string > | splitDottedFields (string k) |
Splits a dotted field and replaces escaped (.) dots with plain dots (.) | |
Public Attributes | |
const | ConstantConflictList = ("name", "struct", "code", "default") |
field keys that conflict with "constant" and "index" | |
const | OptionKeys = ... |
constructor option keys (can be extended by subclassing and reimplementing optionKeys()) | |
const | OutputKeys = ... |
output option keys | |
const | UserOptions = ... |
User options. | |
const | ValidKeys = ... |
default known mapper hash field keys (can be extended by subclassing and reimplementing validKeys()) | |
Private Member Methods | |
checkInputField (string k, string name) | |
raises an error if an invalid input field name is declared; only call this if "input" is defined | |
checkInputProvider () | |
raises an error if no input provider is present | |
checkMap () | |
verifies the input map in the constructor | |
checkMapField (string k, reference< auto > fh) | |
perform per-field pre-processing on the passed map in the constructor More... | |
checkOutputBulkOperation () | |
raises an error if no output provider is present More... | |
checkOutputProvider () | |
raises an error if no output provider is present More... | |
checkTimezoneOption (string rn) | |
verifies a timezone constructor option | |
constructor () | |
private constructor for subclasses | |
convertToHash (int t, string k, reference< auto > fh) | |
convert a field definition to a hash if possible | |
*hash< auto > | doCreateRecordIntern (hash< auto > rec) |
Creates a record with the output data provider. | |
error (string fmt) | |
throws a MAP-ERROR exception; prepends the map name to the description if known More... | |
error2 (string ex, string fmt) | |
throws the given exception; prepends the map name to the description if known | |
fieldLengthError (string k, string val, int ix, int sze, int maxlen, hash< auto > rc) | |
called when a field exceeds its maximum length when processing hashes of lists | |
*AbstractDataProviderType | getExpectedTypeImpl () |
Returns the expected type of data to be submitted, if available. | |
string | getMappingDescription (string key, hash< auto > mapping) |
returns a test description for the input mapping | |
*int | getRecListSize (hash< auto > rec) |
*AbstractDataProviderType | getReturnTypeImpl () |
Returns the type of data that will be returned, if available. | |
hash< auto > | mapDataIntern (hash< auto > rec, *bool do_log_output) |
processes the input record and returns a hash of the mapped values where the keys in the hash returned are the target field names; the order of the fields in the hash returned is the same order as the keys in the map hash. More... | |
nothing | mapFieldIntern (reference< hash< auto >> h, string key, hash< auto > rec, bool do_list, int list_size) |
maps a single field to the target More... | |
auto | mapFieldType (string key, hash< auto > mapping, AbstractDataProviderType type, auto value, hash< auto > rec, auto v) |
performs type handling | |
auto | mapSubclass (hash< auto > m, auto v) |
to be overridden as necessary in subclasses | |
processStructuredInputField (string k, string name, reference< list< string >> struct) | |
raises an error if invalid fields in structured input data are declared; sets "struct" | |
*AbstractDataField | processStructuredOutputField (string k, reference< hash< auto >> fh) |
Raises an error if invalid fields in structured output data are declared; sets "fh". | |
setup (hash< auto > mapv, *hash< auto > opts) | |
sets up the mapper object before checking the mapper hash | |
submitImpl (code enqueue, auto _data) | |
Submits the data for processing by the mapper. More... | |
bool | supportsBulkApiImpl () |
Returns True if the record processor supports bulk operation. More... | |
string | truncateField (string k, string val, int ix, int sze, int maxlen) |
called to truncate fields when processing hashes of lists | |
Static Private Member Methods | |
static *hash< auto > | swapMapperThreadContext (*hash< auto > new_ctx) |
Swap the mapper thread context. | |
Private Attributes | |
const | _Mapper_Thread_Key = "_Mapper_Thread_Key" |
Thread key for mapper thread context while mapping. | |
bool | allow_dot = False |
do not assume struct when field names have a "." in them; instead allow input field names to have a "." in them More... | |
bool | allow_output_dot = False |
do not assume structured/hash output when output field names have a "." in them; instead allow output field names to have a "." in them More... | |
hash< auto > | consth |
map of constant fields | |
int | count = 0 |
count of records mapped | |
const | DeprecatedGlobalTransformOptionMap = ... |
maps deprecated transform options to global tranform options | |
bool | disable_bulk |
disable bulk operations when there is a hash of lists as input when calling mapAuto | |
*hash< auto > | global_submappers |
global submapper hash | |
hash< auto > | global_transform_opts |
global transformation options; can be overridden on a per-field basis | |
hash< auto > | identh |
map of fields to be mapped 1:1 input -> output | |
*list< auto > | identl |
list of fields to be mapped 1:1 input -> output | |
bool | ignore_missing_input |
supresses the output field if the input field is missing and the output is optional | |
*code | info_log |
an optional info logging callback; must accept a sprintf()-style format specifier and optional arguments | |
*hash< string, AbstractDataField > | input |
an optional description of possible input hash keys | |
bool | input_do_request |
indicates that the request/response API should be used for mapper input | |
*code | input_log |
an optional input data logging callback; must accept a hash giving the input data hash | |
*AbstractDataProvider | input_provider |
the optional input data provider | |
*hash< auto > | input_provider_search |
search conditions for the input provider | |
auto | input_request |
the argument for the request/response API for the input provider | |
*hash< auto > | input_request_options |
options for the request when using the request/response API for the input provider | |
bool | input_request_search |
a boolean flag that indicates that an input provider that supports both the record API and the request/response API should use the request search API to provide records | |
*hash< auto > | input_search_options |
search options for the input provider | |
bool | input_suppress_data_provider |
ignore any input data provider; use for type info only | |
*hash< auto > | m_runtime |
current runtime values More... | |
hash | mapc |
the hash providing output field names and mappings More... | |
hash< auto > | mapd |
the hash with a subset of the mappings used dynamically | |
hash< auto > | mapo |
the hash of output records for key order | |
hash< auto > | mapper_handler_context = {} |
mapper handler context hash; to be passed to handlers | |
*hash< auto > | mapper_thread_context |
mapper static thread context hash | |
*string | name |
the optional name for the object (for example a table name); will be prepended to field names in error messages | |
*hash< string, AbstractDataField > | output |
an optional description of the output data structure | |
bool | output_create_ignore_duplicates |
flag indicating that duplicate records should be ignored when creating records with an output mapper | |
bool | output_do_request |
indicates that the request/response API should be used for mapper output | |
*code | output_log |
an optional output data logging callback; must accept a hash giving the output data hash | |
bool | output_nullable = False |
set output fields nullable | |
*AbstractDataProvider | output_provider |
the optional output data provider | |
*AbstractDataProviderBulkOperation | output_provider_bulk_operation |
bulk output object for an output data provider | |
bool | output_provider_passive |
if the output provide should only be used to set the output type | |
bool | output_provider_upsert |
if the upsert operations should be used on the output provider | |
*hash< auto > | output_request_options |
options for the request when using the request/response API for the output provider | |
bool | output_suppress_data_provider |
ignore any output data provider; use for type info only | |
hash< auto > | rconsth |
map of constant runtime fields | |
hash< string, hash< MapperRuntimeKeyInfo > > | runtime_independent_keys |
hash of runtime keys that provide independent mappings (where there is no "requires_role" = "value") | |
hash< string, hash< MapperRuntimeKeyInfo > > | runtime_keys |
map of field keys provided by the "runtime_keys" option | |
hash< string, hash< auto > > | runtime_keys_with_handler |
subset of runtime_keys with handlers | |
bool | structured_input |
flag if the field descriptions were provided to the constructor | |
bool | structured_output |
flag if the field descriptions were provided to the constructor | |
*code | submapper_get |
an optional callback for retrieving a nested mapper; sig: (string name, *hash<auto> opts) {} | |
*code | template_subst |
an optional callback for substituting templated values | |
bool | trunc_all = False |
truncate all strings quietly / automatically to their maximum length | |
const | TypeMap = ... |
maps standard types to auto types to avoid type stripping | |
hash< string, bool > | valid_keys |
hash of valid keys | |
Private:Internal Member Methods | |
AbstractDataProviderRecordIterator | getInputProviderRecordIterator () |
returns a record iterator for an input provider More... | |
this class is a base class for mapping data; see Mapper Examples for usage examples
|
private |
perform per-field pre-processing on the passed map in the constructor
k | the field name |
fh | a reference to the field's value in the map; will be converted to a hash |
|
private |
raises an error if no output provider is present
MAPPER-OUTPUT-BULK-ERROR | no output bulk operation is in progress |
|
private |
raises an error if no output provider is present
MAPPER-OUTPUT-PROVIDER-ERROR | no output provider available in this mapper |
Mapper::Mapper::commit | ( | ) |
Commits data written to the output data provider if the output data provider supports transaction management.
Has no effect if the output data provider does not support transaction management
MAPPER-OUTPUT-PROVIDER-ERROR | no output provider available in this mapper |
builds the object based on a hash providing field mappings, data constraints, and optionally custom mapping logic
mapv | a hash providing field mappings; each hash key is the name of the output field; each value is either True (meaning no translations are done; the data is copied 1:1) or a hash describing the mapping; see Mapper Specification Format for detailed documentation for this option |
opts | an optional hash of options for the mapper; see Mapper Options for a description of valid mapper options |
MAP-ERROR | the map hash has a logical error (ex: "trunc" key given without "maxlen" , invalid map key) |
Mapper::Mapper::discardOutput | ( | ) |
Discards any buffered data in the output data provider in a bulk operation.
This method should always be called if an error occurs in a bulk output operation
MAPPER-OUTPUT-BULK-ERROR | no output bulk operation is in progress |
|
private |
throws a MAP-ERROR
exception; prepends the map name to the description if known
if this method is subclassed, it must also cause an exception to be thrown
Mapper::Mapper::flushOutput | ( | ) |
Flushes any remaining data to the output data provider in a bulk operation.
This method should always be called for successful bulk output operations with an output provider
MAPPER-OUTPUT-BULK-ERROR | no output bulk operation is in progress |
int Mapper::Mapper::getCount | ( | ) |
returns the internal record count
|
private:internal |
returns a record iterator for an input provider
MapperOutputRecordIterator Mapper::Mapper::getOutputIterator | ( | ) |
returns an output record iterator that produces mapped data from the input data provider
MAPPER-INPUT-PROVIDER-ERROR | if no input_provider option was provided in the constructor or the input provider uses the request/response API |
For a possible hash of lists (bulk data) returns the size of the first list found within the hash. If no list if found, returns NOTHING.
rec | a hash representing (possible) bulk data |
auto Mapper::Mapper::getRuntime | ( | string | key | ) |
get current runtime option value for a key
key | the runtime option key |
maps all input records and returns the mapped data as a list of output records
this method applies the mapData() method to all input records and returns the resulting list
recs | a hash of lists of input records |
MISSING-INPUT | a field marked mandatory is missing |
STRING-TOO-LONG | a field value exceeds the maximum value and the 'trunc' key is not set |
INVALID-NUMBER | the field is marked as numeric but the input value contains non-numeric data |
maps all input records and returns the mapped data as a list of output records
this method applies the mapData() method to all input records and returns the resulting list
recs | the list of input records |
MISSING-INPUT | a field marked mandatory is missing |
STRING-TOO-LONG | a field value exceeds the maximum value and the 'trunc' key is not set |
INVALID-NUMBER | the field is marked as numeric but the input value contains non-numeric data |
auto Mapper::Mapper::mapAuto | ( | auto | recs | ) |
maps all input record(s) automatically and returns the mapped data
this method applies the mapAll() method if the input records are given as list and the mapBulk() method if the input is a hash of lists. For a simple hash (one record) it calls the mapData() method.
If NOTHING is passed, then NOTHING is returned with no mapping performed
recs | input records (one record or more), can be represented by list or hash |
MISSING-INPUT | a field marked mandatory is missing |
STRING-TOO-LONG | a field value exceeds the maximum value and the 'trunc' key is not set |
INVALID-NUMBER | the field is marked as numeric but the input value contains non-numeric data |
MAPPER-INPUT-TYPE-ERROR | records input type is not acceptable |
*softlist<hash<auto> > Mapper::Mapper::mapAutoInput | ( | ) |
Runs the input data provider through the mapper and returns the output.
MAPPER-INPUT-PROVIDER-ERROR | no input provider available in this mapper |
maps a set of records in hash of lists format; returns mapped data in a hash of lists format
rec | the input record or record set in case a hash of lists is passed |
crec | an optional simple hash of data to be added to each input row before mapping |
N
rows of a column C
and 'crec = {"C": "mystring"}' then the output will be as if there was 'N' rows with C
= "mystring" on the input.MISSING-INPUT | a field marked mandatory is missing |
STRING-TOO-LONG | a field value exceeds the maximum value and the 'trunc' key is not set |
INVALID-NUMBER | the field is marked as numeric but the input value contains non-numeric data |
processes the input record and returns a hash of the mapped values where the keys in the hash returned are the target field names; the order of the fields in the hash returned is the same order as the keys in the map hash.
rec | the record to translate |
MISSING-INPUT | a field marked mandatory is missing |
STRING-TOO-LONG | a field value exceeds the maximum value and the 'trunc' key is not set |
INVALID-NUMBER | the field is marked as numeric but the input value contains non-numeric data |
processes the input record and returns a hash of the mapped values where the keys in the hash returned are the target field names; the order of the fields in the hash returned is the same order as the keys in the map hash.
rec | the record to translate |
MISSING-INPUT | a field marked mandatory is missing |
STRING-TOO-LONG | a field value exceeds the maximum value and the 'trunc' key is not set |
INVALID-NUMBER | the field is marked as numeric but the input value contains non-numeric data |
|
private |
maps a single field to the target
Performs the actual mapping
h | the hash to be updated with the mapped key/value pair; Depending on the mapper specification and the input, it can contain a bulk record (hash of lists) or a hash of both list and non-list keys (these are considered a "constants" in the output record. |
key | the column name (hash key) to be mapped (target field) |
rec | input record - either single record of hash of lists (batch); to increase performance, the input type (single record vs. batch) is determined by the do_list parameter. In case of bulk input, all the lists are supposed to have the same length list_size |
do_list | - whether the input record rec is single record or bulk format (hash of lists) |
list_size | - size of the lists in case the input is in bulk format (all lists must have the same length, asserted inside the method). |
hash<auto> Mapper::Mapper::optionKeys | ( | ) |
returns a list of valid constructor options for this class (can be overridden in subclasses)
Mapper::Mapper::replaceRuntime | ( | *hash< auto > | runtime | ) |
replaces runtime options
runtime | a hash of runtime options to use to replace the current runtime option hash |
RUNTIME-OPTION-CHANGED | the option has changed during while already processing the data. |
Mapper::Mapper::resetCount | ( | ) |
resets the internal record count
Mapper::Mapper::rollback | ( | ) |
Rolls back data written to the output data provider.
Has no effect if the output data provider does not support transaction management
MAPPER-OUTPUT-PROVIDER-ERROR | no output provider available in this mapper |
Mapper::Mapper::runAutonomous | ( | ) |
Runs the input and output mappers with data providers on each end autonomously.
MAPPER-INPUT-PROVIDER-ERROR | no input provider available in this mapper |
MAPPER-OUTPUT-PROVIDER-ERROR | no output provider available in this mapper |
Mapper::Mapper::setNullableOutput | ( | ) |
sets all fields in the output record as nullable
Mapper::Mapper::setRuntime | ( | hash< auto > | runtime | ) |
adds runtime options to the current runtime option hash
runtime | a hash of runtime options to add to the current runtime option hash |
RUNTIME-OPTION-CHANGED | the option has changed during while already processing the data. |
Mapper::Mapper::setRuntime | ( | string | key, |
auto | value | ||
) |
set the runtime option with "key" to value "value"
key | a string with valid runtime key |
value | anything passed to the current runtime key |
RUNTIME-OPTION-CHANGED | the option has changed during while already processing the data. |
|
private |
Submits the data for processing by the mapper.
enqueue | s closure taking a single arugment that enqueues the processed data for the next step in the pipeline; if no data should be processed onwards, do not call enqueue; if only one record should be processed onwards, then enqueue should be called only once; if multiple records are generated from the input data, then call it once for each generated record; prototype: code enqueue = sub (auto qdata) {}
|
_data | the data to process |
|
private |
returns a list of valid field keys for this class (can be overridden in subclasses)
|
private |
do not assume struct when field names have a "."
in them; instead allow input field names to have a "."
in them
only used when a plain hash is provided for input
|
private |
do not assume structured/hash output when output field names have a "."
in them; instead allow output field names to have a "."
in them
only used when a plain hash is provided for output
|
private |
the hash providing output field names and mappings
note that we must strip types here