SftpPoller::SftpPoller Class Reference

SftpPoller client class implementation. More...

#include <SftpPoller.qm.dox.h>

Public Member Methods
	constructor (Qore::SSH2::SFTPClient n_sftp, hash< auto > nconf)
	creates the SftpPoller object from the SFTPClient argument and configuration hash argument passed More...
	constructor (hash< auto > nconf)
	creates the SftpPoller object from the configuration hash argument passed More...
	destructor ()
	stops the polling operation if in progress and destroys the object
int	getPollCount ()
	returns the current poll count
int	getStoreFile (string remote_path, string local_path, *timeout n_timeout)
	retrieves a remote file and stores it to a local path More...
string	getTextFile (string path, *timeout n_timeout, *string n_encoding)
	retrieves a text file and returns the file's contents
binary	getFile (string path, *timeout n_timeout)
	retrieves a binary file and returns the file's contents
	rename (string old, string nnew, *timeout n_timeout)
	renames a file on the server
	removeFile (string fn, *timeout n_timeout)
	deletes a file on the server
*hash< Qore::SSH2::Ssh2StatInfo >	stat (string path, *timeout n_timeout)
	returns information about a remote file (see Qore::SSH2::SFTPClient::stat() for a description of the return format)
	checkRemotePath (string path, bool write=False, *timeout n_timeout)
	check if a remote path is writable More...
list< hash< SftpPollerFileEventInfo > >	getFiles (int sort=SftpPoller::SortNone, int order=SftpPoller::OrderAsc)
	returns a list of regular file hashes matching any file name mask set for the object More...
int	start ()
	starts polling in the background; returns the thread ID of the polling thread More...
	stopNoWait ()
	stops the polling operation, returns immediately More...
	stop ()
	stops the polling operation, returns when the polling operation has been stopped More...
	waitStop ()
	waits indefinitely for the polling operation to stop; if polling was not in progress then this method returns immediately More...
	startInline ()
	starts the polling operation inline (not in a background thread) More...
bool	runOnce ()
	runs a single poll (useful for checking for errors inline before starting a background thread)
*bool	fileEvent (list< hash< SftpPollerFileEventInfo > > l)
	called for each poll with a list of all files matched before transfer; if this method returns False or nothing, then the singleFileEvent method is not called
abstract	singleFileEvent (hash< SftpPollerFileEventInfo > fih)
	called for each matching file individually whenever matching files are polled with the list of matching file names; if any error occurs here, the error is logged and the polling operation is retried More...
abstract	postSingleFileEvent (hash< SftpPollerFileEventInfo > fih)
	called after singleFileEvent() for each matching file individually whenever matching files are polled with the list of matching file names; if any error occurs here, the polling operation stops More...

Static Public Member Methods
static	checkPath (string path, string type, bool write=False)
	checks a path on the local file system More...

Public Attributes
const	OrderAsc = 0
	ascending sort order
const	OrderDesc = 1
	descending sort order
const	SortNone = 0
	no sorting
const	SortName = 1
	sort by name
const	RequiredKeys = ...
	minimum required keys for all constructors
const	RequiredKeysWithHost = RequiredKeys + "host"
	RequiredKeys for the constructor(hash) without an Qore::SSH2::SFTPClient argument.
const	Defaults
	default values for constructor hash argument
const	OptionalKeys = ...
	optional constructor hash keys
const	AllKeys = RequiredKeysWithHost + Defaults.keys() + OptionalKeys
	all keys
const	ErrorDelay = 1m
	pause when SFTP errors are detected

Private Member Methods
Mutex	m ()
	start mutex
Counter	sc ()
	stop counter
	logInfo (string fmt)
	calls the "log_info" closure or call reference with important information
	logDetail (string fmt)
	calls the "log_detail" closure or call reference with detail information
	logDebug (string fmt)
	calls the "log_debug" closure or call reference with verbose debugging information
	setMask ()
	converts a glob mask into a regex
	sftpSleep (softint secs)
	sleeps for the specificed number of seconds
	run ()
	starts the polling operation

Private Attributes
string	host
	host or address name
int	port
	port
string	user
	user
string	url
	url
*string	pass
	password; one of "pass" or "keyfile" *must* be set
*string	keyfile
	path to the ssh private key in PEM format; one of "pass" or "keyfile" *must* be set
softlist< string >	path = "."
	path(s) to poll
string	rootSftpPath
	path after connect to SFTP server
*string	mask
	file glob name mask (ignored if "regex_mask" also set)
int	poll_interval
	poll interval in seconds
bool	runflag = False
	run flag
bool	get_files
	internal "get files" flag
bool	fatal = False
	internal fatal error flag
int	pollcnt = 0
	internal poll counter
int	tid
	polling tid
timeout	timeout
	timeout in ms
Qore::SSH2::SFTPClient	sftp
	SFTPClient object.
int	reopts = 0
	file matching regex options
*softint	minage
	minimum file age
*string	encoding
	file encoding for text files
*code	log_info
	optional info log closure
*code	log_detail
	optional detail log closure
*code	log_debug
	optional debug log closure
*code	start_thread
	optional start thread closure
*code	sleep
	optional sleep closure
bool	binary
	binary transfer flag (for singleFileEvent())
bool	writable
	chech if path is writable for others in constructor
string	check_file
	name of check writable file
*bool	skip_file_content_retrieval
	whether or not to skip the file content retrieval in runOnce()

Detailed Description

SftpPoller client class implementation.

Member Function Documentation

checkPath()

static SftpPoller::SftpPoller::checkPath ( string  path, string  type, bool  write = False  )

static

checks a path on the local file system

Exceptions

DIR-ERROR

this exception is thrown if the local path does not exist, is not readable, is not a directory, or should be writable and is not

checkRemotePath()

SftpPoller::SftpPoller::checkRemotePath	(	string	path,
		bool	write = False,
		*timeout	n_timeout
	)

check if a remote path is writable

Exceptions

REMOTE-DIR-ERROR

this exception is thrown if the remote path does not exist, is not a directory, or is not not writable

constructor() [1/2]

SftpPoller::SftpPoller::constructor

(

hash< auto >

nconf

)

creates the SftpPoller object from the configuration hash argument passed

Parameters

nconf

a hash with the following keys: host: (required) the hostname or address to connect to port: the integer port number to connect to (default 22; must be > 0 if given) user: the username to use for the connection pass: the password to use for the connection keyfile: the SSH key (file and path to it) path: the remote path(s) for retrieving the files poll_interval: the integer polling interval in seconds (default: 10 seconds; must be > 0 if given) mask: the file glob mask to use (default: "*", ignored if "regex_mask" is also present) regex_mask: a regular expression to use as a mask (overrides any "mask" value) reopts: regular expression match options (ex RE_Caseless for case-insensitive matches) timeout: connection timeout (default 30s) minage: the minimum file age in seconds before a file will be acquired (default: 0) encoding: the encoding for any text files received log_info: a closure or call reference for logging important information; must accept a single string giving the log message log_detail: a closure or call reference for logging detailed information; must accept a single string giving the log message log_debug: a closure or call reference for logging verbose debgugging information; must accept a single string giving the log message start_thread: (required when imported into a context where Qore::PO_NO_THREAD_CONTROL is set) a closure or call reference for starting threads; must return the integer thread ID (if not set then background will be used) sleep: (required when imported into a context where Qore::PO_NO_PROCESS_CONTROL is set) a closure or call reference to use instead of Qore::sleep() (if not set then Qore::sleep() will be used) writable: check if path(s) is/are writable (default: False see also the "check_file" option) check_file: name of a check file. The file will be created and deleted in the path to test if path is writable. If the file alrady exists, it will be deleted. (default: "qore-sftp-check-file") skip_file_content_retrieval: if True then SftpPoller::SftpPoller::singleFileEvent and SftpPoller::SftpPoller::postSingleFileEvent methods won't get 'data' member of the input hash, i.e. the file content won't be read by the SftpPoller class, opening a space for child classes to implement their own file retrieval in SftpPoller::SftpPoller::singleFileEvent or SftpPoller::SftpPoller::postSingleFileEvent (default: False); (since ssh2 1.1)

Exceptions

SFTPPOLLER-CONSTRUCTOR-ERROR	missing required key, invalid port or poll_interval given
SFTPCLIENT-PARAMETER-ERROR	empty hostname passed
SOCKET-CONNECT-ERROR	error establishing socket connection (no listener, port blocked, etc); timeout establishing socket connection
SSH2CLIENT-CONNECT-ERROR	no user name set; ssh2 or libssh2 error
SSH2-ERROR	error initializing or establishing ssh2 session
SSH2CLIENT-AUTH-ERROR	no proper authentication method found
SFTPCLIENT-CONNECT-ERROR	error initializing sftp session or getting remote path

constructor() [2/2]

SftpPoller::SftpPoller::constructor	(	Qore::SSH2::SFTPClient	n_sftp,
		hash< auto >	nconf
	)

creates the SftpPoller object from the SFTPClient argument and configuration hash argument passed

Parameters

n_sftp

the new SFTPClient object

nconf

a hash with the following optional keys: poll_interval: the integer polling interval in seconds (default: 10 seconds; must be > 0 if given) mask: the file glob mask to use (default: "*", ignored if "regex_mask" is also present) path: the remote path(s) for retrieving the files; if a list of strings is given then each path will be polled for matching files according to the "mask" or "regex_mask" option regex_mask: a regular expression to use as a mask (overrides any "mask" value) reopts: regular expression match options (ex RE_Caseless for case-insensitive matches) timeout: connection timeout (default 30s) minage: the minimum file age in seconds before a file will be acquired (default: 0) encoding: the encoding for any text files received binary: if set to True then files are transferred in binary mode by default (with singleFileEvent() usage only), otherwise file data is returned in text format log_info: a closure or call reference for logging important information; must accept a single string giving the log message log_detail: a closure or call reference for logging detailed information; must accept a single string giving the log message log_debug: a closure or call reference for logging verbose debgugging information; must accept a single string giving the log message start_thread: (required when imported into a context where Qore::PO_NO_THREAD_CONTROL is set) a closure or call reference for starting threads; must return the integer thread ID (if not set then background will be used) sleep: (required when imported into a context where Qore::PO_NO_PROCESS_CONTROL is set) a closure or call reference to use instead of Qore::sleep() (if not set then Qore::sleep() will be used) writable: check if path(s) is/are writable (default: False see also the "check_file" option) check_file: name of a check file. The file will be created and deleted in the path to test if path is writable. If the file alrady exists, it will be deleted. (default: "qore-sftp-check-file")

Exceptions

SFTPPOLLER-CONSTRUCTOR-ERROR	missing required key, invalid port or poll_interval given
SFTPCLIENT-PARAMETER-ERROR	empty hostname passed
SOCKET-CONNECT-ERROR	error establishing socket connection (no listener, port blocked, etc); timeout establishing socket connection
SSH2CLIENT-CONNECT-ERROR	no user name set; ssh2 or libssh2 error
SSH2-ERROR	error initializing or establishing ssh2 session
SSH2CLIENT-AUTH-ERROR	no proper authentication method found
SFTPCLIENT-CONNECT-ERROR	error initializing sftp session or getting remote path

getFiles()

list< hash< SftpPollerFileEventInfo > > SftpPoller::SftpPoller::getFiles	(	int	sort = SftpPoller::SortNone,
		int	order = SftpPoller::OrderAsc
	)

returns a list of regular file hashes matching any file name mask set for the object

Parameters

sort	the sort option for the list returned
order	the ordering of sorted data returned

Returns

a list of regular file hashes with the following keys in each list element:

• name: the name of the file, link, or directory
• size: the size of the file in bytes
• uid: the UID of the owner of the file
• gid: the GID of the owner of the file
• mode: the permissions / mode of the file
• atime: the last accessed date/time of the file
• mtime: the last modified date/time of the file
• type: the type of file is always "REGULAR"
• perm: a string giving UNIX-style permissions for the file (ex: "-rwxr-xr-x")

getStoreFile()

int SftpPoller::SftpPoller::getStoreFile	(	string	remote_path,
		string	local_path,
		*timeout	n_timeout
	)

retrieves a remote file and stores it to a local path

Parameters

remote_path	the remote file path
local_path	the local file path
n_timeout	a timeout in milliseconds

Returns

the number of bytes transferred

postSingleFileEvent()

abstract SftpPoller::SftpPoller::postSingleFileEvent ( hash< SftpPollerFileEventInfo >  fih )

pure virtual

called after singleFileEvent() for each matching file individually whenever matching files are polled with the list of matching file names; if any error occurs here, the polling operation stops

This method would normally delete / rename / move files processed by singleFileEvent() so that they would not be polled a second time. If an error occurs in this operation, then the polling event will stop since continuing after failing to delete, rename, or move a file already processed would cause the file to be processed more than once.

Parameters

fih

a hash of file data and information with the following keys: name: the name of the file, link, or directory size: the size of the file in bytes uid: the UID of the owner of the file gid: the GID of the owner of the file mode: the permissions / mode of the file atime: the last accessed date/time of the file mtime: the last modified date/time of the file type: the type of file; one of: "REGULAR", "DIRECTORY", "SYMBOLIC-LINK", "BLOCK-DEVICE", "CHARACTER-DEVICE", "FIFO", "SYMBOLIC-LINK", "SOCKET", or "UNKNOWN" perm: a string giving UNIX-style permissions for the file (ex: "-rwxr-xr-x") data: the file's data; this will be a string unless the "binary" option is set to True, in which case this key is assigned to the files binary data; this hash key is only present if skip_file_content_retrieval was False in the SftpPoller::SftpPoller::constructor options filepath: the remote filepath relative to SFTP root directory

singleFileEvent()

abstract SftpPoller::SftpPoller::singleFileEvent ( hash< SftpPollerFileEventInfo >  fih )

pure virtual

called for each matching file individually whenever matching files are polled with the list of matching file names; if any error occurs here, the error is logged and the polling operation is retried

Parameters

fih

a hash of file data and information with the following keys: name: the name of the file, link, or directory size: the size of the file in bytes uid: the UID of the owner of the file gid: the GID of the owner of the file mode: the permissions / mode of the file atime: the last accessed date/time of the file mtime: the last modified date/time of the file type: the type of file; one of: "REGULAR", "DIRECTORY", "SYMBOLIC-LINK", "BLOCK-DEVICE", "CHARACTER-DEVICE", "FIFO", "SYMBOLIC-LINK", "SOCKET", or "UNKNOWN" perm: a string giving UNIX-style permissions for the file (ex: "-rwxr-xr-x") data: the file's data; this will be a string unless the "binary" option is set to True, in which case this key is assigned to the files binary data; this hash key is only present if skip_file_content_retrieval was False in the SftpPoller::SftpPoller::constructor options filepath: the remote filepath relative to SFTP root directory

start()

int SftpPoller::SftpPoller::start

(

)

starts polling in the background; returns the thread ID of the polling thread

if polling had already been started, then the thread ID of the polling thread is returned immediately

startInline()

SftpPoller::SftpPoller::startInline

(

)

starts the polling operation inline (not in a background thread)

Exceptions

SFTPPOLLER-ERROR

this exception is thrown if polling is already in progress

stop()

SftpPoller::SftpPoller::stop

(

)

stops the polling operation, returns when the polling operation has been stopped

if polling was not in progress then this method returns immediately

Exceptions

THREAD-ERROR

this exception is thrown if this method is called from the event thread since it would result in a deadlock

See also

stopNoWait()

stopNoWait()

SftpPoller::SftpPoller::stopNoWait

(

)

stops the polling operation, returns immediately

See also

stop()

waitStop()

SftpPoller::SftpPoller::waitStop

(

)

waits indefinitely for the polling operation to stop; if polling was not in progress then this method returns immediately

Exceptions

THREAD-ERROR

this exception is thrown if this method is called from the event thread since it would result in a deadlock

---

The documentation for this class was generated from the following file:

• doxygen/qlib/SftpPoller/SftpPoller.qm.dox.h