FtpPoller client class implementation. More...

#include <FtpPoller.qm.dox.h>

Public Member Methods
	constructor (hash< auto > nconf)
	creates the FtpPoller object from the configuration hash argument passed

	constructor (Qore::FtpClient n_ftp, hash< auto > nconf)
	creates the FtpPoller object from the FtpClient argument and configuration hash argument passed

	destructor ()
	stops the polling operation if in progress and destroys the object

*bool	fileEvent (list< hash< FtpPollerFileEventInfo > > l)
	called for each poll with a list of all files matched before transfer

binary	getFile (string path)
	retrieves a binary file and returns the file's contents

list< hash< FtpPollerFileEventInfo > >	getFiles (string subdir, int sort=FtpPoller::SortNone, int order=FtpPoller::OrderAsc)
	returns a list of regular file hashes matching any file name mask set for the object

int	getPollCount ()
	returns the current poll count

hash< FtpPollerFileEventInfo >	getRemoteFileData (hash< FtpPollerFileEventInfo > event)
	Retrieves remote file data and adds it to the event data.

	getStoreFile (string remote_path, string local_path)
	retrieves a remote file and stores it to a local path

string	getTextFile (string path)
	retrieves a text file and returns the file's contents

abstract	postSingleFileEvent (hash< FtpPollerFileEventInfo > event)
	called after singleFileEvent() for each matching file individually

	removeFile (string fn)
	deletes a file on the server

	rename (string old, string nnew)
	renames a file on the server

bool	runOnce ()
	runs a single poll (useful for checking for errors inline before starting a background thread)

abstract	singleFileEvent (hash< FtpPollerFileEventInfo > event)
	called for each matching file individually whenever matching files are polled

int	start ()
	starts polling in the background; returns the thread ID of the polling thread

	startInline ()
	starts the polling operation inline (not in a background thread)

	stop ()
	stops the polling operation, returns when the polling operation has been stopped

	stopNoWait ()
	stops the polling operation, returns immediately

	updateOptions (FtpClient ftp)
	Updates options.

	waitStop ()
	waits indefinitely for the polling operation to stop

Static Public Member Methods
static	checkPath (string path, string type, bool write=False)
	checks a path on the local file system

Public Attributes
const	AllKeys = RequiredKeysWithHost + keys Defaults + OptionalKeys
	all keys

const	Defaults = ...
	default values for constructor hash argument

const	ErrorDelay = 1m
	pause when Ftp errors are detected

const	OptionalKeys = ...
	optional constructor hash keys

const	OrderAsc = 0
	ascending sort order

const	OrderDesc = 1
	descending sort order

const	RequiredKeys = ...
	minimum required keys for all constructors

const	RequiredKeysWithHost = RequiredKeys + "host"
	RequiredKeys for the constructor(hash) without an Qore::FtpClient argument.

const	SortName = 1
	sort by name

const	SortNone = 0
	no sorting

Private Member Methods
	ftpSleep (softint secs)
	sleeps for the specificed number of seconds

	logDebug (string fmt,...)
	calls the `"log_debug"` closure or call reference with verbose debugging information

	logDetail (string fmt,...)
	calls the `"log_detail"` closure or call reference with detail information

	logInfo (string fmt,...)
	calls the `"log_info"` closure or call reference with important information

Mutex	m ()
	start mutex

hash< FtpPollerFileEventInfo >	retrieveFile (hash< FtpPollerFileEventInfo > event)
	Retrieves the remote file to local_dir directly.

hash< FtpPollerFileEventInfo >	retrieveTempFile (hash< FtpPollerFileEventInfo > event)
	Retrieves the remote file to local_dir using a temporary file.

	run ()
	starts the polling operation

Counter	sc ()
	stop counter

	setMask ()
	converts a glob mask into a regex

Private Attributes
bool	atomic_transfer
	Atomic transfer flag for use with `local_dir`.

bool	delete_remote_file
	Delete file on remote FTP server after processing?

bool	fatal = False
	internal fatal error flag

Qore::FtpClient	ftp
	FtpClient object.

bool	get_files
	internal "get files" flag

string	host
	host or address name

string	local_dir
	Local directory to transfer file.

*code	log_debug
	optional debug log closure

*code	log_detail
	optional detail log closure

*code	log_info
	optional info log closure

*string	mask
	file glob name mask (ignored if `"regex_mask"` also set)

*softint	minage
	minimum file age

*string	move_remote_file
	Remote target directory for moving the file on the remote FTP server after processing.

FtpClient	new_ftp
	Connection to update while polling.

*string	pass
	password

softlist< string >	path = "."
	path(s) to poll

int	poll_interval
	poll interval in seconds

int	pollcnt = 0
	internal poll counter

int	port
	port

string	protocol
	Protocol ("ftp" or "ftps")

int	reopts = 0
	file matching regex options

string	rootFtpPath
	path after connecting to Ftp server

bool	runflag = False
	run flag

*code	sleep
	optional sleep closure

*code	start_thread
	optional start thread closure

string	tempfile_template
	The temporary filename template when `local_dir` is set.

int	tid
	polling tid

timeout	timeout
	timeout in ms

string	url
	url

hash< UrlInfo >	urlh
	url hash

string	user
	user

Detailed Description

FtpPoller client class implementation.

See the constructor options for information on how to configure this object; note that either

delete_remote_file or
move_remote_file must be set, or FTP files polled will remain where they are in the remote FTP server, unless they are removed or moved during polling processing.

Only one of these options may be set at one time; setting both will result in an error

Member Function Documentation

◆ checkPath()

static FtpPoller::FtpPoller::checkPath	(	string	path,
		string	type,
		bool	write = `False`
	)

static

checks a path on the local file system

Exceptions

DIR-ERROR this exception is thrown if the local path does not exist, is not readable, is not a directory, or should be writable and is not

◆ constructor() [1/2]

FtpPoller::FtpPoller::constructor ( hash< auto > nconf )

creates the FtpPoller object from the configuration hash argument passed

Parameters

nconf

a hash with the following keys:

atomic_transfer: if True and local_dir is set, then tempfile_template is used to write FTP files to a temporary filename during transfer
delete_remote_file: if True then the remote file is deleted on the FTP server after processing. Cannot be set if move_remote_file is also set
host: (required) the hostname or address to connect to
local_dir: a local directory to use to transfer remote files so large file data does not appear in events
log_debug: a closure or call reference for logging verbose debgugging information; must accept a single string giving the log message
log_detail: a closure or call reference for logging detailed information; must accept a single string giving the log message
log_info: a closure or call reference for logging important information; must accept a single string giving the log message
mask: the file glob mask to use (default: "*", ignored if "regex_mask" is also present)
minage: the minimum file age in seconds before a file will be acquired (default: 0)
move_remote_file: the remote target directory on the FTP server where files will be moved to after processing; cannot be set if delete_remote_file is also set
pass: the password to use for the connection
path: the remote path(s) for retrieving the files
poll_interval: the integer polling interval in seconds (default: 10 seconds; must be > 0 if given)
port: the integer port number to connect to (default 21; must be > 0 if given)
protocol: either "ftp" (the detault) or "ftps" for secure FTP
regex_mask: a regular expression to use as a mask (overrides any "mask" value)
reopts: regular expression match options (ex RE_Caseless for case-insensitive matches)
start_thread: (required when imported into a context where Qore::PO_NO_THREAD_CONTROL is set) a closure or call reference for starting threads; must return the integer thread ID (if not set then Background Operator (background) will be used)
sleep: (required when imported into a context where Qore::PO_NO_PROCESS_CONTROL is set) a closure or call reference to use instead of Qore::sleep() (if not set then Qore::sleep() will be used)
tempfile_template: (used when local_dir and atomic_transfer are set) the temporary filename prefix to use when transferring files to the local directory specified by local_dir; use "%s" as the placeholder for the target filename
user: the username to use for the connection

Exceptions

FTPPOLLER-CONSTRUCTOR-ARG-ERROR	missing required key, invalid port, poll_interval, or protocol given
FTPCLIENT-PARAMETER-ERROR	empty hostname passed
SOCKET-CONNECT-ERROR	error establishing socket connection (no listener, port blocked, etc)
FTPCLIENT-CONNECT-ERROR	error initializing ftp session or getting remote path

◆ constructor() [2/2]

FtpPoller::FtpPoller::constructor	(	Qore::FtpClient	n_ftp,
		hash< auto >	nconf
	)

creates the FtpPoller object from the FtpClient argument and configuration hash argument passed

Parameters

n_ftp the new FtpClient object

nconf

a hash with the following optional keys:

atomic_transfer: if True and local_dir is set, then tempfile_template is used to write FTP files to a temporary filename during transfer
delete_remote_file: if True then the remote file is deleted on the FTP server after processing. Cannot be set if move_remote_file is also set
local_dir: a local directory to use to transfer remote files so large file data does not appear in events
log_debug: a closure or call reference for logging verbose debgugging information; must accept a single string giving the log message
log_detail: a closure or call reference for logging detailed information; must accept a single string giving the log message
log_info: a closure or call reference for logging important information; must accept a single string giving the log message
mask: the file glob mask to use (default: "*", ignored if "regex_mask" is also present)
minage: the minimum file age in seconds before a file will be acquired (default: 0)
move_remote_file: the remote target directory on the FTP server where files will be moved to after processing; cannot be set if delete_remote_file is also set
path: the remote path(s) for retrieving the files; if a list of strings is given then each path will be polled for matching files according to the "mask" or "regex_mask" option
poll_interval: the integer polling interval in seconds (default: 10 seconds; must be > 0 if given)
regex_mask: a regular expression to use as a mask (overrides any "mask" value)
reopts: regular expression match options (ex RE_Caseless for case-insensitive matches)
sleep: (required when imported into a context where Qore::PO_NO_PROCESS_CONTROL is set) a closure or call reference to use instead of Qore::sleep() (if not set then Qore::sleep() will be used)
start_thread: (required when imported into a context where Qore::PO_NO_THREAD_CONTROL is set) a closure or call reference for starting threads; must return the integer thread ID (if not set then Background Operator (background) will be used)
tempfile_template: (used when local_dir and atomic_transfer are set) the temporary filename prefix to use when transferring files to the local directory specified by local_dir; use "%s" as the placeholder for the target filename

Note: Either delete_remote_file or move_remote_file must be set, or FTP files polled will remain where they are in the remote FTP server, unless they are removed or moved during polling processing. Only one of these options may be set at one time; setting both will result in an error

Exceptions

FTPPOLLER-CONSTRUCTOR-ARG-ERROR	error with poller options
FTPCLIENT-PARAMETER-ERROR	empty hostname passed
SOCKET-CONNECT-ERROR	error establishing socket connection (no listener, port blocked, etc); timeout establishing socket connection
FTPCLIENT-CONNECT-ERROR	error initializing ftp session or getting remote path

◆ fileEvent()

*bool FtpPoller::FtpPoller::fileEvent ( list< hash< FtpPollerFileEventInfo > > l )

called for each poll with a list of all files matched before transfer

if this method returns False or NOTHING, then the singleFileEvent method is not called

◆ getFiles()

list< hash< FtpPollerFileEventInfo > > FtpPoller::FtpPoller::getFiles	(	string	subdir,
		int	sort = `FtpPoller::SortNone`,
		int	order = `FtpPoller::OrderAsc`
	)

returns a list of regular file hashes matching any file name mask set for the object

Parameters

subdir	the directory to check
sort	the sort option for the list returned
order	the ordering of sorted data returned

Returns

a list of regular file hashes with the following keys in each list element:

name: the name of the file, link, or directory
size: the size of the file in bytes
mtime: the last modified date/time of the file

◆ getStoreFile()

FtpPoller::FtpPoller::getStoreFile	(	string	remote_path,
		string	local_path
	)

retrieves a remote file and stores it to a local path

Parameters

remote_path	the remote file path
local_path	the local file path

◆ postSingleFileEvent()

abstract FtpPoller::FtpPoller::postSingleFileEvent ( hash< FtpPollerFileEventInfo > event )

called after singleFileEvent() for each matching file individually

If any error occurs here, the polling operation stops

This method would normally delete / rename / move files processed by singleFileEvent() so that they would not be polled a second time. If an error occurs in this operation, then the polling event will stop since continuing after failing to delete, rename, or move a file already processed would cause the file to be processed more than once.

Parameters

event a hash of file data

◆ singleFileEvent()

abstract FtpPoller::FtpPoller::singleFileEvent ( hash< FtpPollerFileEventInfo > event )

called for each matching file individually whenever matching files are polled

If any error occurs here, the error is logged and the polling operation is retried

Parameters

event a hash of file data

◆ start()

int FtpPoller::FtpPoller::start ( )

starts polling in the background; returns the thread ID of the polling thread

if polling had already been started, then the thread ID of the polling thread is returned immediately

◆ startInline()

FtpPoller::FtpPoller::startInline ( )

starts the polling operation inline (not in a background thread)

Exceptions

FTPPOLLER-ERROR this exception is thrown if polling is already in progress

◆ stop()

FtpPoller::FtpPoller::stop ( )

stops the polling operation, returns when the polling operation has been stopped

if polling was not in progress then this method returns immediately

Exceptions

THREAD-ERROR this exception is thrown if this method is called from the event thread since it would result in a deadlock

See also: stopNoWait()

◆ stopNoWait()

FtpPoller::FtpPoller::stopNoWait ( )

stops the polling operation, returns immediately

See also: stop()

◆ waitStop()

FtpPoller::FtpPoller::waitStop ( )

waits indefinitely for the polling operation to stop

If polling was not in progress then this method returns immediately

Exceptions

THREAD-ERROR this exception is thrown if this method is called from the event thread since it would result in a deadlock

Member Data Documentation

◆ delete_remote_file

bool FtpPoller::FtpPoller::delete_remote_file

private

Delete file on remote FTP server after processing?

This cannot be True while move_remote_file is set

◆ move_remote_file

*string FtpPoller::FtpPoller::move_remote_file

private

Remote target directory for moving the file on the remote FTP server after processing.

This cannot be True while delete_remote_file is set

Public Member Methods

Static Public Member Methods

Public Attributes

Private Member Methods

Private Attributes

Detailed Description

Member Function Documentation

◆ checkPath()

◆ constructor() [1/2]

◆ constructor() [2/2]

◆ fileEvent()

◆ getFiles()

◆ getStoreFile()

◆ postSingleFileEvent()

◆ singleFileEvent()

◆ start()

◆ startInline()

◆ stop()

◆ stopNoWait()

◆ waitStop()

Member Data Documentation

◆ delete_remote_file

◆ move_remote_file