Module Description

Table of Contents

• Module Overview
• Binary Modules
• User Modules
  • User Module Declarations
  • The "public" Keyword
  • Module Example
  • Program Scope in Object-Oriented Programs Using User Modules Providing Their Own Threads

Module Overview

Qore modules allow the Qore language to be extended at run-time.

To load a module at parse time (normally required for most modules), use the %requires or %try-module parse directive. If the named feature is not already present in Qore, Qore will try to load the module from its list of module directories (module path). This list contains the following directories:

1. directories listed in the environment variable QORE_MODULE_DIR (see Environment Variables)
2. USER_MODULE_VER_DIR - build variable set to directory where Qore user modules of the specific Qore version should be installed (ex: /usr/share/qore-modules/0.8.12)
3. MODULE_VER_DIR - build variable set to directory where Qore binary modules of the specific Qore version should be installed (ex: /usr/lib/x86_64-linux-gnu/qore-modules/0.8.12)
4. USER_MODULE_DIR - build variable set to directory where user modules of external binary modules should be installed (ex: /usr/share/qore-modules)
5. MODULE_DIR - build variable set to directory where external binary modules should be installed (ex: /usr/lib/x86_64-linux-gnu/qore-modules)
6. directories added using %append-module-path

Modules can also be loaded directly by using a path relative to the Qore script or module requiring them when using %requires or %try-module parse directives:

%requires ../../directory/Module.qm

Module implementation could be separated to multiple files and stored as a directory named with the same name as the module. To require them use %requires ModuleName without ".qm" extension. All files with the extenstion *qc, *ql (e.g. functions and classes) inside the directory will be automatically included at program parse time.

Examlpe of such separated module:

ModuleName/
    |-ModuleName.qm         # typical "qore" module file with the exception that implementation of classes and functions that it is used is stored separately
    |-SomeModuleClass.qc
    |-SomeModuleFunction.ql

Use the load_module() function to load Qore modules at run-time; however, note that any module providing parse support (classes, constants, functions, etc) must be loaded at parse time using the %requires or %try-module parse directive.

From Qore 0.7.1 onwards, you can specify a comparison operator (one of < <=, =, >=, or >) and version information after the module name as well. Version numbers are compared via integer comparisons of each element, where elements are separated by a '.'. If one of the versions does not have as many elements as another, the missing elements are assumed to be '0' (i.e. version "1.0" compared with version "1.0.1" will be extended to "1.0.0").

Also note that DBI drivers are loaded on demand by the Qore::SQL::Datasource and Qore::SQL::DatasourcePool classes.

There are two types of modules: Binary Modules and User Modules.

At the time of writing this documentation, the following modules exist for Qore:

Modules Provided With Qore

Type	Module	Description
binary	astparser	Provides a language parser API to Qore
user	BulkSqlUtil	Provides APIs based on SqlUtil for executing bulk DML operations
user	ConnectionProvider	Provides an API for pluggable connection and URL providers
user	CsvUtil	Provides code to help parse CSV or other structured text files and also to easily generate such files
user	DatasourceProvider	Provides an API to translate datasource identifiers to datasource connection strings
user	DebugCmdLine	Provides an API to develop command line debugger clients
user	DebugLinenoiseCmdLine	Provides an API for command line debugger clients based on the linenoise module
user	DebugProgramControl	Provides an API to develop debug server
user	DebugHandler	Provides a websocket for debug server
user	DebugUtil	Provides a general stuff for debugger
user	FilePoller	Provides an API for polling files on the filesystem
user	FixedLengthUtil	Provides code to help parse data in format of fixed length lines records
user	FreetdsSqlUtil	Provides a high-level DB-independent API for working with MS SQL Server and Sybase database objects; loaded automatically by the SqlUtil module when working with the freetds DBI driver for automated schema management, programmatic DB access, schema and data synchronization, and more
user	FsUtil	Provides a high level API for working with the filesystem
user	HttpServer	Provides a multi-threaded HTTP server and request handler APIs; provides the infrastructure for server-side HTTP services
user	HttpServerUtil	Provides base definitions for handler classes integrating with the HttpServer module
user	Logger	Provides a flexible logging API (aka "log4q")
user	MailMessage	Provides supporting classes for the Pop3Client and SmtpClient modules; mail message serialization and deserialization, attachment handling
user	Mapper	Provides data mapping classes and interators
user	Mime	Provides MIME definitions and functions; MIME type lookups, MIME encoding and decoding functions, MIME multipart handling, etc
user	MysqlSqlUtil	Provides a high-level DB-independent API for working with MySQL database objects; loaded automatically by the SqlUtil module when working with MySQL databases for automated schema management, programmatic DB access, schema and data synchronization, and more
user	OracleSqlUtil	Provides a high-level DB-independent API for working with Oracle database objects; loaded automatically by the SqlUtil module when working with Oracle databases for automated schema management, programmatic DB access, schema and data synchronization, and more
user	Pop3Client	Provides POP3 client functionality; provides an API to retrieve email messages from a POP3 server
user	PgsqlSqlUtil	Provides a high-level DB-independent API for working with PostgreSQL database objects; loaded automatically by the SqlUtil module when working with PostgreSQL databases for automated schema management, programmatic DB access, schema and data synchronization, and more
user	Qdx	Provides support for documentation generation with doxygen from Qore sources
user	Qorize	Provides basic support for automatically-generating Qore code from data
user	QUnit	Provides an API for defining and executing tests and reporting results in various formats
binary	reflection	Provides a reflection API to Qore
user	RestClient	Provides a simple API for communicating with HTTP servers implementing REST services
user	RestHandler	Provides an easy to use interface to the Qore HttpServer module for implementing server-side REST services
user	RestSchemaValidator	Provides a REST schema validation API
user	SalesforceRestClient	Provides an API for communicating with Salesforce.com REST services (extends and reexports functionality provided by the RestClient module)
user	Schema	Provides automatic schema management functionality as a meta-layer for SqlUtil's medium and low-level schema management functionality
user	SewioRestClient	providing APIs for communicating with Sewio.net's RTLS Studio REST API
user	SewioWebSocketClient	providing APIs for communicating with Sewio.net's RTLS Studio WebSocket API
user	SmtpClient	Provides SMTP client functionality; provides an API for sending emails via an SMTP server
user	SqlUtil	Provides a high-level DB-independent API for working with databases; for automated schema management, programmatic DB access, schema and data synchronization, and more
user	Swagger	Provides a Swagger 2.0 REST API validation API
user	TableMapper	Provides a data mapping functionality and iterator support using SqlUtil and Mapper to map arbitrary data to an SQL table target
user	TelnetClient	Provides Telnet client functionality
user	TextWrap	Provides text wrapping and filling functionality
user	Util	Provides a some miscellaneous generally useful routines; often used by other user modules for example
user	WebSocketClient	Provides an event-driven client API for connecting to WebSocket servers
user	WebSocketHandler	Provides an interface to the Qore HttpServer module for implementing server-side WebSocket services
user	WebSocketUtil	Provides common client and server code for implementing WebSocket protocol services in Qore
user	WebUtil	Provides higher-level classes for implementing more complex web services in Qore; works with the HttpServer module
user	ZeyosRestClient	providing APIs for communicating with Zeyos.com's REST API

Modules Provided Separately

Type	Module	Description
binary	freetds	Provides a FreeTDS-based DBI driver for communicating with Sybase and MS SQL Server databases
binary	fsevent	Provides an event-driven filesystem event API
binary	jni	Provides access to Java APIs in Qore
binary	json	Provides JSON and JSON-RPC client functionality, also provides the following user modules: - JsonRpcHandler: provides infrastructure for implementing JSON-RPC server-side services using the HttpServer module
binary	linenoise	Provides a readline-like API to Qore under a permissive license
binary	msgpack	Provides APIs for efficient data serialization in Qore using the MessagePack format
binary	mysql	Provides a MySQL / MariaDB / Percona DBI driver
binary	odbc	Provides an ODBC DBI driver to Qore
binary	oracle	Provides an Oracle DBI driver, providing many advanced features such as support for named types and collections, advanced queuing, etc, also provides the following user modules: - OracleExtensions: provides infrastructure for SQL tracing
binary	openldap	Provides an OpenLDAP API to Qore
binary	pgsql	Provides a PostgreSQL DBI driver
binary	process	Provides an API for controlling external processes
binary	sqlite3	Provides an SQLite3 DBI driver
binary	ssh2	Provides SSH2 and SFTP functionality, also provides the following user module: - SftpPoller: provides event-driven support for polling an SFTP server for new data
binary	sybase	Provides a Sybase DBI driver
binary	sysconf	Provides sysconf, pathconf, and confstr APIs
binary	tibae	Provides TIBCO ActiveEnterprise(TM) (TIBCO, Inc) functionality
binary	tibrv	Provides TIBCO Rendezvous(R) (TIBCO, Inc) functionality
binary	uuid	Provices an API for generating UUIDs
binary	xml	Provides XML (SAX and DOM parsers), XPath, XML-RPC, SOAP client and server, etc functionality, also provides the following user modules: - SoapClient: provides an easy to use API for making requests to SOAP servers - SalesforceSoapClient: provides an easy to use API for making requests to Salesforce.com SOAP servers - SoapHandler: provides infrastructure for implementing SOAP server-side services using the HttpServer module - WSDL: provides underlying web service/WSDL support for the SoapClient and SoapHandler modules - XmlRpcHandler: provides infrastructure for implementing XML-RPC server-side services using the HttpServer module
binary	xmlsec	Provides xmldsig and xmlenc functionality
binary	yaml	Provides YAML functionality, also provides the following user modules: - DataStreamClient: provides a DataStream client API extending the RestClient - DataStreamRequestHandler: provides a DataStream server-side handler API extending the RestHandler - DataStreamUtil: provides underlying DataStream client and server protocol support - YamlRpcClient: provides an API for easily making YAML-RPC calls over the network - YamlRpcHandler: provides infrastructure for implementing YAML-RPC server-side services using the HttpServer module

Binary Modules

Binary modules are written in C++ and delivered in binary form. They must conform to the Qore Module API and have the file extension "*.qmod". Binary modules normally depend on other shared libraries and therefore can only be loaded if the libraries they require are present on the system and can be found in the library path.

Binary modules are merged into Program objects in the same way as the static system namespace objects are imported. It is possible to import a binary module and be able to use only part of its functionality, for example, if the importing Program cannot access the filesystem, and the module has functions that access the filesystem, only the functions that access the filesystem will not be available. In user modules, the functional domain is set on the module level and not on the individual function or method level as with builtin objects, so user modules are either completely imported or not at all.

Please note that as of version 0.7.0 onwards, the source code for binary modules has been split from the main Qore library source code into separate projects; see Qore's home page for more information.

User Modules

User modules are written in Qore and delivered in source form. They must have the extension "*.qm".

User modules will have their own dedicated Program object; the Program object is created automatically when the module is loaded and initialized. The module's Program object is created with locked parse options as follows:

• PO_NO_TOP_LEVEL_STATEMENTS: modules provide API services and are not designed to be executed at the top-level. Put any initialization code in the module's init closure
• PO_REQUIRE_PROTOTYPES: to ensure that module's APIs are transparent, all method and function declarations must include parameter and return types
• PO_REQUIRE_OUR: this parse option is meant to ensure that typos in variable names are caught at parse time and to ensure transparency of variable types in the module's source code
• PO_IN_MODULE: this parse option is set so that module code is recognized by the system when parsing modules

When parsing module code, the default warning mask is set to WARN_MODULES, and any warnings are treated as errors. Furthermore, any restrictions that the importing Program object has will be added to the module's Program object.

Any namespaces, classes, constants, functions, and global variables declared as public will be exported into the importing Program object; all other declarations will be private to the module's Program object.

Note

The root namespace of a module Program is always public by default, however declarations in the root namespace must be marked public to be exported to importing Program objects.

If a user module imports other modules with the requires(reexport) form of the %requires parse directive, then any loaded definitions will also be loaded into the importing Program.

Note that global variables exported from a module's Program object are exported as references; each global variable declared in a module is unique, and they reside in the module's Program object.

Furthermore, when using an environment with multiple Program objects, if a user module has already been loaded and initialized, then it's functional domain mask is compared against any importing Program object's restrictions; if the module uses functionality that is not allowed in the importing Program object, then an exception is raised and the module is not imported.

Also note that the Qore::Program::constructor() applies a mask to the parse option mask option when Program objects are created in a user module; in this case the parse options passed to the child Program object are masked with the current parse options in the user module, and they are locked so that they cannot be made less restrictive. This is to prevent user modules from circumventing functional restrictions imposed by parse options.

User Module Declarations

User modules are declared with a special syntax in Qore:

User Module Declaration Syntax
    module name {
        version = "version string";
        desc = "description string";
        author = "author string";
        [url = "URL string";]
        [license = "license string";]
        [init = initialization closure;]
        [del = deletion closure;]
    }

Module properties are as follows:

• version: (required) must be assigned to a string giving the version of the module
• desc: (required) must be assigned to a string giving a text description of the module
• author: (required) must be assigned to a string giving the module's author's name
• url: (optional) if present, must be a string giving the URL of the module
• license: (optional) if present, must be a string giving the license of the module
• init: (optional) if present, must be a closure that will be executed when the module is loaded; this can be used to initialize the module, for example
• del: (optional) if present, must be a closure that will be executed when the module is unloaded; this can be used to stop running services or free resources managed and still allocated by the module, for example

Note

• any unhandled exceptions in the init closure will cause the module to fail to load
• unhandled exceptions in the init and del closures are displayed on stdout

The "public" Keyword

Only objects defined with the public keyword are made available in Program objects importing the user module. All other declarations and definitions are private to the module.

The public keyword also affects inheritance in child Program objects as well as controlling exported symbols from user modules.

The public keyword must be used with the following declarations in the module to export them:

• namespaces: namespaces themselves must be declared public in order for any of their contents also to be exported (it is an error to try to declare public members of a module-private namespace). ex:

  public namespace MyNamespace { ... }

• classes: classes not declared public will not be exported; there is no way to export part of a class; either the entire class is exported or it is not. ex:

  public MyNamespace {
      public MyClass { ... }
  }

• constants: constants must be declared public to be exported; ex:

  public namespace MyNamespace {
      public const MyConst = 100;
  }

• functions: function variants must be declared public in order to be exported. ex:

  public MyNamespace {
      public int sub my_func() { ... }
  }

• global variables: only global variables declared public will be exported. ex:

  public namespace MyNamespace {
      public our int OurInt;
  }

Note

Global variable declarations in a namespace declaration cannot be initialized at the point they are declared, also, since Qore::PO_NO_TOP_LEVEL_STATEMENTS is set for user module Program objects, global variables also cannot be initialized at the top-level. Use the init closure to initialize the module and any global variables requiring initialization.

User modules are only imported into a Program if the importing Program can use all of the capabilities used in the user module. It is not possible to partially import a user module (in contrast with binary modules, which can be imported even if they expose functionality not allowed in the importing Program, however that functionality will not be available in that case). User module Program objects have a functional domain attribute set on the Program level, so either a user module is imported in its entirety or not at all.

Module Example

Here is an example declaring user module "foo" version "1.0":

module foo {
    version = "1.0";
    desc = "test module";
    author = "Foobar Industries, inc";
    url = "http://example.com";
    license = "MIT";

    init = sub () {
        Foo::initialized = True;
        Bar::OurBool = False;
    };

    del = sub () {
        print("goodbye, cruel world!\n");
    }
}

# do not use "$", assume local scope for variables
%new-style

# nothing in namespace ::Foo is exported
namespace Foo {
    # inline global variable declarations cannot be initialized when declared
    our bool initialized;

    class SuperClass {
    }
    class NotSoGreatClass {
    }
    class UnstableClass {
    }
}

# public members of namespace ::Bar are exported
public namespace Bar {
    # Bar::SomeClass is exported
    public class SomeClass {
    }

    # Bar::add(int, int) is exported
    public int sub add(int x, int y) { return x + y; }

    # Bar::OurBool is exported
    public our bool OurBool;

    # Bar::PrivateClass is not exported
    class PrivateClass {
    }
}

Since

Qore 0.8.4 user modules are supported

Qore 0.9.0 user modules implementation could be separated to multiple files

Program Scope in Object-Oriented Programs Using User Modules Providing Their Own Threads

When using the %exec-class parse directive, the application object will go out of scope as soon as the constructor terminates (unless there are valid scope-extending references to the application object, such as making an assignment of self to a global variable in the contructor).

Therefore when using a module that provides services in its own threads (such as, for example, the HttpServer module), it's important to make sure that the application object does not go out of scope while non-static method call references to the application object are needed by active threads in the user module (for example, non-static method call references passed as callbacks to the HttpServer module, etc).

This also applies to call references to non-static methods passed to set_signal_handler().

If a module thread tries to use a callback or closure in a Program that has already gone out of scope, an OBJECT-ALREADY-DELETED exception ("attempt to access member of an already-deleted object") is thrown.

Therefore in such cases it's best to wait for all threads in any modules to terminate before allowing the application object's constructor to terminate.

For an example of this, see the example program examples/httpserver.q where the main Program calls HttpServer::waitStop() before exiting the application object's constructor for this reason.