Qore FtpPoller Module Reference  1.0
FtpPoller::FtpPoller Class Referenceabstract

FtpPoller client class implementation. More...

Public Member Methods

 constructor (hash< auto > nconf)
 creates the FtpPoller object from the configuration hash argument passed More...
 
 constructor (Qore::FtpClient n_ftp, hash< auto > nconf)
 creates the FtpPoller object from the FtpClient argument and configuration hash argument passed More...
 
 destructor ()
 stops the polling operation if in progress and destroys the object
 
*bool fileEvent (list l)
 called for each poll with a list of all files matched before transfer; if this method returns False or NOTHING, then the singleFileEvent method is not called
 
binary getFile (string path)
 retrieves a binary file and returns the file's contents
 
list< hash< FtpPollerFileEventInfo > > getFiles (string subdir, int sort=FtpPoller::SortNone, int order=FtpPoller::OrderAsc)
 returns a list of regular file hashes matching any file name mask set for the object More...
 
int getPollCount ()
 returns the current poll count
 
 getStoreFile (string remote_path, string local_path)
 retrieves a remote file and stores it to a local path More...
 
string getTextFile (string path)
 retrieves a text file and returns the file's contents
 
abstract postSingleFileEvent (hash< FtpPollerFileEventInfo > fih)
 called after singleFileEvent() for each matching file individually whenever matching files are polled with the list of matching file names; if any error occurs here, the polling operation stops More...
 
 removeFile (string fn)
 deletes a file on the server
 
 rename (string old, string nnew)
 renames a file on the server
 
bool runOnce ()
 runs a single poll (useful for checking for errors inline before starting a background thread)
 
abstract singleFileEvent (hash< FtpPollerFileEventInfo > fih)
 called for each matching file individually whenever matching files are polled with the list of matching file names; if any error occurs here, the error is logged and the polling operation is retried More...
 
int start ()
 starts polling in the background; returns the thread ID of the polling thread More...
 
 startInline ()
 starts the polling operation inline (not in a background thread) More...
 
 stop ()
 stops the polling operation, returns when the polling operation has been stopped More...
 
 stopNoWait ()
 stops the polling operation, returns immediately More...
 
 waitStop ()
 waits indefinitely for the polling operation to stop; if polling was not in progress then this method returns immediately More...
 

Static Public Member Methods

static checkPath (string path, string type, bool write=False)
 checks a path on the local file system More...
 

Public Attributes

const AllKeys = RequiredKeysWithHost + Defaults.keys() + OptionalKeys
 all keys
 
const Defaults
 default values for constructor hash argument
 
const ErrorDelay = 1m
 pause when Ftp errors are detected
 
const OptionalKeys = ...
 optional constructor hash keys
 
const OrderAsc = 0
 ascending sort order
 
const OrderDesc = 1
 descending sort order
 
const RequiredKeys = ...
 minimum required keys for all constructors
 
const RequiredKeysWithHost = RequiredKeys + "host"
 RequiredKeys for the constructor(hash) without an Qore::FtpClient argument.
 
const SortName = 1
 sort by name
 
const SortNone = 0
 no sorting
 

Private Member Methods

 ftpSleep (softint secs)
 sleeps for the specificed number of seconds
 
 logDebug (string fmt)
 calls the "log_debug" closure or call reference with verbose debugging information
 
 logDetail (string fmt)
 calls the "log_detail" closure or call reference with detail information
 
 logInfo (string fmt)
 calls the "log_info" closure or call reference with important information
 
Mutex m ()
 start mutex
 
 run ()
 starts the polling operation
 
Counter sc ()
 stop counter
 
 setMask ()
 converts a glob mask into a regex
 

Private Attributes

bool binary
 binary transfer flag (for singleFileEvent())
 
*string encoding
 file encoding for text files
 
bool fatal = False
 internal fatal error flag
 
Qore::FtpClient ftp
 FtpClient object.
 
bool get_files
 internal "get files" flag
 
string host
 host or address name
 
*code log_debug
 optional debug log closure
 
*code log_detail
 optional detail log closure
 
*code log_info
 optional info log closure
 
*string mask
 file glob name mask (ignored if "regex_mask" also set)
 
*softint minage
 minimum file age
 
*string pass
 password
 
softlist< stringpath = "."
 path(s) to poll
 
int poll_interval
 poll interval in seconds
 
int pollcnt = 0
 internal poll counter
 
int port
 port
 
int reopts = 0
 file matching regex options
 
string rootFtpPath
 path after connecting to Ftp server
 
bool runflag = False
 run flag
 
*code sleep
 optional sleep closure
 
*code start_thread
 optional start thread closure
 
int tid
 polling tid
 
timeout timeout
 timeout in ms
 
string url
 url
 
hash< UrlInfo > urlh
 url hash
 
string user
 user
 

Detailed Description

FtpPoller client class implementation.

Member Function Documentation

◆ checkPath()

static FtpPoller::FtpPoller::checkPath ( string  path,
string  type,
bool  write = False 
)
static

checks a path on the local file system

Exceptions
DIR-ERRORthis exception is thrown if the local path does not exist, is not readable, is not a directory, or should be writable and is not

◆ constructor() [1/2]

FtpPoller::FtpPoller::constructor ( hash< auto >  nconf)

creates the FtpPoller object from the configuration hash argument passed

Parameters
nconfa hash with the following keys:
  • host: (required) the hostname or address to connect to
  • port: the integer port number to connect to (default 21; must be > 0 if given)
  • user: the username to use for the connection
  • pass: the password to use for the connection
  • path: the remote path(s) for retrieving the files
  • poll_interval: the integer polling interval in seconds (default: 10 seconds; must be > 0 if given)
  • mask: the file glob mask to use (default: "*", ignored if "regex_mask" is also present)
  • regex_mask: a regular expression to use as a mask (overrides any "mask" value)
  • reopts: regular expression match options (ex RE_Caseless for case-insensitive matches)
  • minage: the minimum file age in seconds before a file will be acquired (default: 0)
  • encoding: the encoding for any text files received
  • log_info: a closure or call reference for logging important information; must accept a single string giving the log message
  • log_detail: a closure or call reference for logging detailed information; must accept a single string giving the log message
  • log_debug: a closure or call reference for logging verbose debgugging information; must accept a single string giving the log message
  • start_thread: (required when imported into a context where Qore::PO_NO_THREAD_CONTROL is set) a closure or call reference for starting threads; must return the integer thread ID (if not set then Background Operator (background) will be used)
  • sleep: (required when imported into a context where Qore::PO_NO_PROCESS_CONTROL is set) a closure or call reference to use instead of Qore::sleep() (if not set then Qore::sleep() will be used)
Exceptions
FTPPOLLER-CONSTRUCTOR-ERRORmissing required key, invalid port or poll_interval given
FTPCLIENT-PARAMETER-ERRORempty hostname passed
SOCKET-CONNECT-ERRORerror establishing socket connection (no listener, port blocked, etc)
SSH2CLIENT-CONNECT-ERRORno user name set; ssh2 or libssh2 error
SSH2-ERRORerror initializing or establishing ssh2 session
SSH2CLIENT-AUTH-ERRORno proper authentication method found
FTPCLIENT-CONNECT-ERRORerror initializing ftp session or getting remote path

◆ constructor() [2/2]

FtpPoller::FtpPoller::constructor ( Qore::FtpClient  n_ftp,
hash< auto >  nconf 
)

creates the FtpPoller object from the FtpClient argument and configuration hash argument passed

Parameters
n_ftpthe new FtpClient object
nconfa hash with the following optional keys:
  • poll_interval: the integer polling interval in seconds (default: 10 seconds; must be > 0 if given)
  • mask: the file glob mask to use (default: "*", ignored if "regex_mask" is also present)
  • path: the remote path(s) for retrieving the files; if a list of strings is given then each path will be polled for matching files according to the "mask" or "regex_mask" option
  • regex_mask: a regular expression to use as a mask (overrides any "mask" value)
  • reopts: regular expression match options (ex RE_Caseless for case-insensitive matches)
  • minage: the minimum file age in seconds before a file will be acquired (default: 0)
  • encoding: the encoding for any text files received
  • binary: if set to True then files are transferred in binary mode by default (with singleFileEvent() usage only), otherwise file data is returned in text format
  • log_info: a closure or call reference for logging important information; must accept a single string giving the log message
  • log_detail: a closure or call reference for logging detailed information; must accept a single string giving the log message
  • log_debug: a closure or call reference for logging verbose debgugging information; must accept a single string giving the log message
  • start_thread: (required when imported into a context where Qore::PO_NO_THREAD_CONTROL is set) a closure or call reference for starting threads; must return the integer thread ID (if not set then Background Operator (background) will be used)
  • sleep: (required when imported into a context where Qore::PO_NO_PROCESS_CONTROL is set) a closure or call reference to use instead of Qore::sleep() (if not set then Qore::sleep() will be used)
Exceptions
FTPPOLLER-CONSTRUCTOR-ERRORmissing required key, invalid port or poll_interval given
FTPCLIENT-PARAMETER-ERRORempty hostname passed
SOCKET-CONNECT-ERRORerror establishing socket connection (no listener, port blocked, etc); timeout establishing socket connection
SSH2CLIENT-CONNECT-ERRORno user name set; ssh2 or libssh2 error
SSH2-ERRORerror initializing or establishing ssh2 session
SSH2CLIENT-AUTH-ERRORno proper authentication method found
FTPCLIENT-CONNECT-ERRORerror initializing ftp session or getting remote path

◆ getFiles()

list<hash<FtpPollerFileEventInfo> > FtpPoller::FtpPoller::getFiles ( string  subdir,
int  sort = FtpPoller::SortNone,
int  order = FtpPoller::OrderAsc 
)

returns a list of regular file hashes matching any file name mask set for the object

Parameters
subdirthe directory to check
sortthe sort option for the list returned
orderthe ordering of sorted data returned
Returns
a list of regular file hashes with the following keys in each list element:
  • name: the name of the file, link, or directory
  • size: the size of the file in bytes
  • mtime: the last modified date/time of the file

◆ getStoreFile()

FtpPoller::FtpPoller::getStoreFile ( string  remote_path,
string  local_path 
)

retrieves a remote file and stores it to a local path

Parameters
remote_paththe remote file path
local_paththe local file path

◆ postSingleFileEvent()

abstract FtpPoller::FtpPoller::postSingleFileEvent ( hash< FtpPollerFileEventInfo fih)
pure virtual

called after singleFileEvent() for each matching file individually whenever matching files are polled with the list of matching file names; if any error occurs here, the polling operation stops

This method would normally delete / rename / move files processed by singleFileEvent() so that they would not be polled a second time. If an error occurs in this operation, then the polling event will stop since continuing after failing to delete, rename, or move a file already processed would cause the file to be processed more than once.

Parameters
fiha hash of file data and information with the following keys:
  • name: the name of the file, link, or directory
  • size: the size of the file in bytes
  • mtime: the last modified date/time of the file
  • data: the file's data; this will be a string unless the "binary" option is set to True, in which case this key is assigned to the files binary data
  • filepath: the remote filepath relative to FTP root directory

◆ singleFileEvent()

abstract FtpPoller::FtpPoller::singleFileEvent ( hash< FtpPollerFileEventInfo fih)
pure virtual

called for each matching file individually whenever matching files are polled with the list of matching file names; if any error occurs here, the error is logged and the polling operation is retried

Parameters
fiha hash of file data and information with the following keys:
  • name: the name of the file, link, or directory
  • size: the size of the file in bytes
  • mtime: the last modified date/time of the file
  • data: the file's data; this will be a string unless the "binary" option is set to True, in which case this key is assigned to the files binary data
  • filepath: the remote filepath relative to FTP root directory

◆ start()

int FtpPoller::FtpPoller::start ( )

starts polling in the background; returns the thread ID of the polling thread

if polling had already been started, then the thread ID of the polling thread is returned immediately

◆ startInline()

FtpPoller::FtpPoller::startInline ( )

starts the polling operation inline (not in a background thread)

Exceptions
FTPPOLLER-ERRORthis exception is thrown if polling is already in progress

◆ stop()

FtpPoller::FtpPoller::stop ( )

stops the polling operation, returns when the polling operation has been stopped

if polling was not in progress then this method returns immediately

Exceptions
THREAD-ERRORthis exception is thrown if this method is called from the event thread since it would result in a deadlock
See also
stopNoWait()

◆ stopNoWait()

FtpPoller::FtpPoller::stopNoWait ( )

stops the polling operation, returns immediately

See also
stop()

◆ waitStop()

FtpPoller::FtpPoller::waitStop ( )

waits indefinitely for the polling operation to stop; if polling was not in progress then this method returns immediately

Exceptions
THREAD-ERRORthis exception is thrown if this method is called from the event thread since it would result in a deadlock