Qore FtpPoller Module Reference 2.0
Loading...
Searching...
No Matches
FtpPoller::FtpPoller Class Referenceabstract

FtpPoller client class implementation. More...

#include <FtpPoller.qm.dox.h>

Public Member Methods

 constructor (hash< auto > nconf)
 creates the FtpPoller object from the configuration hash argument passed More...
 
 constructor (Qore::FtpClient n_ftp, hash< auto > nconf)
 creates the FtpPoller object from the FtpClient argument and configuration hash argument passed More...
 
 destructor ()
 stops the polling operation if in progress and destroys the object
 
*bool fileEvent (list< hash< FtpPollerFileEventInfo > > l)
 called for each poll with a list of all files matched before transfer More...
 
binary getFile (string path)
 retrieves a binary file and returns the file's contents
 
list< hash< FtpPollerFileEventInfo > > getFiles (string subdir, int sort=FtpPoller::SortNone, int order=FtpPoller::OrderAsc)
 returns a list of regular file hashes matching any file name mask set for the object More...
 
int getPollCount ()
 returns the current poll count
 
hash< FtpPollerFileEventInfogetRemoteFileData (hash< FtpPollerFileEventInfo > event)
 Retrieves remote file data and adds it to the event data.
 
 getStoreFile (string remote_path, string local_path)
 retrieves a remote file and stores it to a local path More...
 
string getTextFile (string path)
 retrieves a text file and returns the file's contents
 
abstract postSingleFileEvent (hash< FtpPollerFileEventInfo > event)
 called after singleFileEvent() for each matching file individually More...
 
 removeFile (string fn)
 deletes a file on the server
 
 rename (string old, string nnew)
 renames a file on the server
 
bool runOnce ()
 runs a single poll (useful for checking for errors inline before starting a background thread)
 
abstract singleFileEvent (hash< FtpPollerFileEventInfo > event)
 called for each matching file individually whenever matching files are polled More...
 
int start ()
 starts polling in the background; returns the thread ID of the polling thread More...
 
 startInline ()
 starts the polling operation inline (not in a background thread) More...
 
 stop ()
 stops the polling operation, returns when the polling operation has been stopped More...
 
 stopNoWait ()
 stops the polling operation, returns immediately More...
 
 waitStop ()
 waits indefinitely for the polling operation to stop More...
 

Static Public Member Methods

static checkPath (string path, string type, bool write=False)
 checks a path on the local file system More...
 

Public Attributes

const AllKeys = RequiredKeysWithHost + keys Defaults + OptionalKeys
 all keys
 
const Defaults
 default values for constructor hash argument
 
const ErrorDelay = 1m
 pause when Ftp errors are detected
 
const OptionalKeys = ...
 optional constructor hash keys
 
const OrderAsc = 0
 ascending sort order
 
const OrderDesc = 1
 descending sort order
 
const RequiredKeys = ...
 minimum required keys for all constructors
 
const RequiredKeysWithHost = RequiredKeys + "host"
 RequiredKeys for the constructor(hash) without an Qore::FtpClient argument.
 
const SortName = 1
 sort by name
 
const SortNone = 0
 no sorting
 

Private Member Methods

 ftpSleep (softint secs)
 sleeps for the specificed number of seconds
 
 logDebug (string fmt)
 calls the "log_debug" closure or call reference with verbose debugging information
 
 logDetail (string fmt)
 calls the "log_detail" closure or call reference with detail information
 
 logInfo (string fmt)
 calls the "log_info" closure or call reference with important information
 
Mutex m ()
 start mutex
 
hash< FtpPollerFileEventInforetrieveFile (hash< FtpPollerFileEventInfo > event)
 Retrieves the remote file to local_dir directly.
 
hash< FtpPollerFileEventInforetrieveTempFile (hash< FtpPollerFileEventInfo > event)
 Retrieves the remote file to local_dir using a temporary file.
 
 run ()
 starts the polling operation
 
Counter sc ()
 stop counter
 
 setMask ()
 converts a glob mask into a regex
 

Private Attributes

bool atomic_transfer
 Atomic transfer flag for use with local_dir.
 
bool fatal = False
 internal fatal error flag
 
Qore::FtpClient ftp
 FtpClient object.
 
bool get_files
 internal "get files" flag
 
string host
 host or address name
 
string local_dir
 Local directory to transfer file.
 
*code log_debug
 optional debug log closure
 
*code log_detail
 optional detail log closure
 
*code log_info
 optional info log closure
 
*string mask
 file glob name mask (ignored if "regex_mask" also set)
 
*softint minage
 minimum file age
 
*string pass
 password
 
softlist< string > path = "."
 path(s) to poll
 
int poll_interval
 poll interval in seconds
 
int pollcnt = 0
 internal poll counter
 
int port
 port
 
string protocol
 Protocol ("ftp" or "ftps")
 
int reopts = 0
 file matching regex options
 
string rootFtpPath
 path after connecting to Ftp server
 
bool runflag = False
 run flag
 
*code sleep
 optional sleep closure
 
*code start_thread
 optional start thread closure
 
string tempfile_template
 The temporary filename template when local_dir is set.
 
int tid
 polling tid
 
timeout timeout
 timeout in ms
 
string url
 url
 
hash< UrlInfo > urlh
 url hash
 
string user
 user
 

Detailed Description

FtpPoller client class implementation.

Member Function Documentation

◆ checkPath()

static FtpPoller::FtpPoller::checkPath ( string  path,
string  type,
bool  write = False 
)
static

checks a path on the local file system

Exceptions
DIR-ERRORthis exception is thrown if the local path does not exist, is not readable, is not a directory, or should be writable and is not

◆ constructor() [1/2]

FtpPoller::FtpPoller::constructor ( hash< auto >  nconf)

creates the FtpPoller object from the configuration hash argument passed

Parameters
nconfa hash with the following keys:
  • atomic_transfer: if True and local_dir is set, then tempfile_template is used to write FTP files to a temporary filename during transfer
  • host: (required) the hostname or address to connect to
  • local_dir: a local directory to use to transfer remote files so large file data does not appear in events
  • log_debug: a closure or call reference for logging verbose debgugging information; must accept a single string giving the log message
  • log_detail: a closure or call reference for logging detailed information; must accept a single string giving the log message
  • log_info: a closure or call reference for logging important information; must accept a single string giving the log message
  • mask: the file glob mask to use (default: "*", ignored if "regex_mask" is also present)
  • minage: the minimum file age in seconds before a file will be acquired (default: 0)
  • pass: the password to use for the connection
  • path: the remote path(s) for retrieving the files
  • poll_interval: the integer polling interval in seconds (default: 10 seconds; must be > 0 if given)
  • port: the integer port number to connect to (default 21; must be > 0 if given)
  • protocol: either "ftp" (the detault) or "ftps" for secure FTP
  • regex_mask: a regular expression to use as a mask (overrides any "mask" value)
  • reopts: regular expression match options (ex RE_Caseless for case-insensitive matches)
  • start_thread: (required when imported into a context where Qore::PO_NO_THREAD_CONTROL is set) a closure or call reference for starting threads; must return the integer thread ID (if not set then Background Operator (background) will be used)
  • sleep: (required when imported into a context where Qore::PO_NO_PROCESS_CONTROL is set) a closure or call reference to use instead of Qore::sleep() (if not set then Qore::sleep() will be used)
  • tempfile_template: (used when local_dir and atomic_transfer are set) the temporary filename prefix to use when transferring files to the local directory specified by local_dir; use "%s" as the placeholder for the target filename
  • user: the username to use for the connection
Exceptions
FTPPOLLER-CONSTRUCTOR-ERRORmissing required key, invalid port, poll_interval, or protocol given
FTPCLIENT-PARAMETER-ERRORempty hostname passed
SOCKET-CONNECT-ERRORerror establishing socket connection (no listener, port blocked, etc)
SSH2CLIENT-CONNECT-ERRORno user name set; ssh2 or libssh2 error
SSH2-ERRORerror initializing or establishing ssh2 session
SSH2CLIENT-AUTH-ERRORno proper authentication method found
FTPCLIENT-CONNECT-ERRORerror initializing ftp session or getting remote path

◆ constructor() [2/2]

FtpPoller::FtpPoller::constructor ( Qore::FtpClient  n_ftp,
hash< auto >  nconf 
)

creates the FtpPoller object from the FtpClient argument and configuration hash argument passed

Parameters
n_ftpthe new FtpClient object
nconfa hash with the following optional keys:
  • atomic_transfer: if True and local_dir is set, then tempfile_template is used to write FTP files to a temporary filename during transfer
  • local_dir: a local directory to use to transfer remote files so large file data does not appear in events
  • log_debug: a closure or call reference for logging verbose debgugging information; must accept a single string giving the log message
  • log_detail: a closure or call reference for logging detailed information; must accept a single string giving the log message
  • log_info: a closure or call reference for logging important information; must accept a single string giving the log message
  • mask: the file glob mask to use (default: "*", ignored if "regex_mask" is also present)
  • minage: the minimum file age in seconds before a file will be acquired (default: 0)
  • path: the remote path(s) for retrieving the files; if a list of strings is given then each path will be polled for matching files according to the "mask" or "regex_mask" option
  • poll_interval: the integer polling interval in seconds (default: 10 seconds; must be > 0 if given)
  • regex_mask: a regular expression to use as a mask (overrides any "mask" value)
  • reopts: regular expression match options (ex RE_Caseless for case-insensitive matches)
  • sleep: (required when imported into a context where Qore::PO_NO_PROCESS_CONTROL is set) a closure or call reference to use instead of Qore::sleep() (if not set then Qore::sleep() will be used)
  • start_thread: (required when imported into a context where Qore::PO_NO_THREAD_CONTROL is set) a closure or call reference for starting threads; must return the integer thread ID (if not set then Background Operator (background) will be used)
  • tempfile_template: (used when local_dir and atomic_transfer are set) the temporary filename prefix to use when transferring files to the local directory specified by local_dir; use "%s" as the placeholder for the target filename
Exceptions
FTPPOLLER-CONSTRUCTOR-ERRORmissing required key, invalid port or poll_interval given
FTPCLIENT-PARAMETER-ERRORempty hostname passed
SOCKET-CONNECT-ERRORerror establishing socket connection (no listener, port blocked, etc); timeout establishing socket connection
SSH2CLIENT-CONNECT-ERRORno user name set; ssh2 or libssh2 error
SSH2-ERRORerror initializing or establishing ssh2 session
SSH2CLIENT-AUTH-ERRORno proper authentication method found
FTPCLIENT-CONNECT-ERRORerror initializing ftp session or getting remote path

◆ fileEvent()

*bool FtpPoller::FtpPoller::fileEvent ( list< hash< FtpPollerFileEventInfo > >  l)

called for each poll with a list of all files matched before transfer

if this method returns False or NOTHING, then the singleFileEvent method is not called

◆ getFiles()

list< hash< FtpPollerFileEventInfo > > FtpPoller::FtpPoller::getFiles ( string  subdir,
int  sort = FtpPoller::SortNone,
int  order = FtpPoller::OrderAsc 
)
inline

returns a list of regular file hashes matching any file name mask set for the object

Parameters
subdirthe directory to check
sortthe sort option for the list returned
orderthe ordering of sorted data returned
Returns
a list of regular file hashes with the following keys in each list element:
  • name: the name of the file, link, or directory
  • size: the size of the file in bytes
  • mtime: the last modified date/time of the file

◆ getStoreFile()

FtpPoller::FtpPoller::getStoreFile ( string  remote_path,
string  local_path 
)

retrieves a remote file and stores it to a local path

Parameters
remote_paththe remote file path
local_paththe local file path

◆ postSingleFileEvent()

abstract FtpPoller::FtpPoller::postSingleFileEvent ( hash< FtpPollerFileEventInfo event)
pure virtual

called after singleFileEvent() for each matching file individually

If any error occurs here, the polling operation stops

This method would normally delete / rename / move files processed by singleFileEvent() so that they would not be polled a second time. If an error occurs in this operation, then the polling event will stop since continuing after failing to delete, rename, or move a file already processed would cause the file to be processed more than once.

Parameters
eventa hash of file data

◆ singleFileEvent()

abstract FtpPoller::FtpPoller::singleFileEvent ( hash< FtpPollerFileEventInfo event)
pure virtual

called for each matching file individually whenever matching files are polled

If any error occurs here, the error is logged and the polling operation is retried

Parameters
eventa hash of file data

◆ start()

int FtpPoller::FtpPoller::start ( )

starts polling in the background; returns the thread ID of the polling thread

if polling had already been started, then the thread ID of the polling thread is returned immediately

◆ startInline()

FtpPoller::FtpPoller::startInline ( )

starts the polling operation inline (not in a background thread)

Exceptions
FTPPOLLER-ERRORthis exception is thrown if polling is already in progress

◆ stop()

FtpPoller::FtpPoller::stop ( )

stops the polling operation, returns when the polling operation has been stopped

if polling was not in progress then this method returns immediately

Exceptions
THREAD-ERRORthis exception is thrown if this method is called from the event thread since it would result in a deadlock
See also
stopNoWait()

◆ stopNoWait()

FtpPoller::FtpPoller::stopNoWait ( )

stops the polling operation, returns immediately

See also
stop()

◆ waitStop()

FtpPoller::FtpPoller::waitStop ( )

waits indefinitely for the polling operation to stop

If polling was not in progress then this method returns immediately

Exceptions
THREAD-ERRORthis exception is thrown if this method is called from the event thread since it would result in a deadlock