\hypertarget{character_encoding_character_encoding_overview}{}\doxysection{Overview}\label{character_encoding_character_encoding_overview}
The Qore language is character-\/encoding aware. All strings are assumed to have the default character encoding, unless the program explicitly specified another encoding for certain objects and operations. Every Qore string has a character encoding ID attached to it, so, when another encoding is required, the Qore language will attempt to do an encoding translation.

Qore uses the operating system\textquotesingle{}s {\ttfamily iconv} library functions to perform any encoding conversions.

Qore supports character encodings that are backwards compatible with 7-\/bit {\ttfamily A\+S\+C\+II}. This includes all {\ttfamily I\+S\+O-\/8859-\/$\ast$} character encodings, {\ttfamily U\+T\+F-\/8}, {\ttfamily K\+O\+I\+R-\/8}, {\ttfamily K\+O\+I\+U-\/8}, and {\ttfamily K\+O\+I7}, among others (see the table below\+: \mbox{\hyperlink{character_encoding_known_encodings}{Known Character Encodings}}).

However, mutibyte character encodings are currently only properly supported for {\ttfamily U\+T\+F-\/8}. For {\ttfamily U\+T\+F-\/8} strings, the \mbox{\hyperlink{group__string__functions_ga5ae69f518a47e6146bf2285a54804455}{length()}}, \mbox{\hyperlink{group__string__functions_gadaea9cfa0f698a6fb0650f70cf8d929e}{index()}}, \mbox{\hyperlink{group__string__functions_ga55663008aa59489b869e558b29c7c22d}{rindex()}}, \mbox{\hyperlink{group__string__functions_gafe621446ada382d5b086a1d2a20934c9}{substr()}}, \mbox{\hyperlink{group__string__functions_ga1a77feb53e86b659be6d9eea7d3618e8}{reverse()}}, the \mbox{\hyperlink{operators_splice}{splice operator}}, \mbox{\hyperlink{group__string__functions_string_formatting}{print formatting}} (regarding field lengths) functions and methods taking format strings, and regular expression operators and functions, all work with character offsets, which may be different than byte offsets. For all character encodings other than {\ttfamily U\+T\+F-\/8}, a 1 byte=1 character relationship is assumed.

Qore will accept any encoding name given to it, even if it is not a known encoding name or alias. In this case, Qore will tag the strings with this encoding, and pass this user-\/defined encoding name to the {\ttfamily iconv} library when encodings must be converted. This allows programmers to use encodings known by the system\textquotesingle{}s {\ttfamily iconv} library, but unknown to Qore. In this case, Qore will assume that the strings are backwards compatible with {\ttfamily A\+S\+C\+II}, meaning that that one character is represented by one byte and that the strings are null-\/terminated.

Note that when Qore matches an encoding name to a code or alias in the following table, the comparison is not case-\/sensitive.\hypertarget{character_encoding_known_encodings}{}\doxysection{Character Encodings Known to Qore}\label{character_encoding_known_encodings}
 \tabulinesep=1mm
\begin{longtabu}spread 0pt [c]{*{3}{|X[-1]}|}
\hline
{\bfseries{Code}} &{\bfseries{Aliases}} &{\bfseries{Description}}  \\\cline{1-3}
{\ttfamily I\+S\+O-\/8859-\/1}  &{\ttfamily I\+S\+O88591}, {\ttfamily I\+S\+O8859-\/1}, {\ttfamily I\+S\+O-\/88591}, {\ttfamily I\+S\+O8859\+P1}, {\ttfamily I\+S\+O81}, {\ttfamily L\+A\+T\+I\+N1}, {\ttfamily L\+A\+T\+I\+N-\/1}  &latin-\/1, Western European character set  \\\cline{1-3}
{\ttfamily I\+S\+O-\/8859-\/2}  &{\ttfamily I\+S\+O88592}, {\ttfamily I\+S\+O8859-\/2}, {\ttfamily I\+S\+O-\/88592}, {\ttfamily I\+S\+O8859\+P2}, {\ttfamily I\+S\+O82}, {\ttfamily L\+A\+T\+I\+N2}, {\ttfamily L\+A\+T\+I\+N-\/2}  &latin-\/2, Central European character set  \\\cline{1-3}
{\ttfamily I\+S\+O-\/8859-\/3}  &{\ttfamily I\+S\+O88593}, {\ttfamily I\+S\+O8859-\/3}, {\ttfamily I\+S\+O-\/88593}, {\ttfamily I\+S\+O8859\+P3}, {\ttfamily I\+S\+O83}, {\ttfamily L\+A\+T\+I\+N3}, {\ttfamily L\+A\+T\+I\+N-\/3}  &latin-\/3, Southern European character set  \\\cline{1-3}
{\ttfamily I\+S\+O-\/8859-\/4}  &{\ttfamily I\+S\+O88594}, {\ttfamily I\+S\+O8859-\/4}, {\ttfamily I\+S\+O-\/88594}, {\ttfamily I\+S\+O8859\+P4}, {\ttfamily I\+S\+O84}, {\ttfamily L\+A\+T\+I\+N4}, {\ttfamily L\+A\+T\+I\+N-\/4}  &latin-\/4, Northern European character set  \\\cline{1-3}
{\ttfamily I\+S\+O-\/8859-\/5}  &{\ttfamily I\+S\+O88595}, {\ttfamily I\+S\+O8859-\/5}, {\ttfamily I\+S\+O-\/88595}, {\ttfamily I\+S\+O8859\+P5}, {\ttfamily I\+S\+O85}  &Cyrillic character set  \\\cline{1-3}
{\ttfamily I\+S\+O-\/8859-\/6}  &{\ttfamily I\+S\+O88596}, {\ttfamily I\+S\+O8859-\/6}, {\ttfamily I\+S\+O-\/88596}, {\ttfamily I\+S\+O8859\+P6}, {\ttfamily I\+S\+O86}  &Arabic character set  \\\cline{1-3}
{\ttfamily I\+S\+O-\/8859-\/7}  &{\ttfamily I\+S\+O88597}, {\ttfamily I\+S\+O8859-\/7}, {\ttfamily I\+S\+O-\/88597}, {\ttfamily I\+S\+O8859\+P7}, {\ttfamily I\+S\+O87}  &Greek character set  \\\cline{1-3}
{\ttfamily I\+S\+O-\/8859-\/8}  &{\ttfamily I\+S\+O88598}, {\ttfamily I\+S\+O8859-\/8}, {\ttfamily I\+S\+O-\/88598}, {\ttfamily I\+S\+O8859\+P8}, {\ttfamily I\+S\+O88}  &Hebrew character set  \\\cline{1-3}
{\ttfamily I\+S\+O-\/8859-\/9}  &{\ttfamily I\+S\+O88599}, {\ttfamily I\+S\+O8859-\/9}, {\ttfamily I\+S\+O-\/88599}, {\ttfamily I\+S\+O8859\+P9}, {\ttfamily I\+S\+O89}, {\ttfamily L\+A\+T\+I\+N5}, {\ttfamily L\+A\+T\+I\+N-\/5}  &latin-\/5, Turkish character set  \\\cline{1-3}
{\ttfamily I\+S\+O-\/8859-\/10}  &{\ttfamily I\+S\+O885910}, {\ttfamily I\+S\+O8859-\/10}, {\ttfamily I\+S\+O-\/885910}, {\ttfamily I\+S\+O8859\+P10}, {\ttfamily I\+S\+O810}, {\ttfamily L\+A\+T\+I\+N6}, {\ttfamily L\+A\+T\+I\+N-\/6}  &latin-\/6, Nordic character set  \\\cline{1-3}
{\ttfamily I\+S\+O-\/8859-\/11}  &{\ttfamily I\+S\+O885911}, {\ttfamily I\+S\+O8859-\/11}, {\ttfamily I\+S\+O-\/885911}, {\ttfamily I\+S\+O8859\+P11}, {\ttfamily I\+S\+O811}  &Thai character set  \\\cline{1-3}
{\ttfamily I\+S\+O-\/8859-\/13}  &{\ttfamily I\+S\+O885913}, {\ttfamily I\+S\+O8859-\/13}, {\ttfamily I\+S\+O-\/885913}, {\ttfamily I\+S\+O8859\+P13}, {\ttfamily I\+S\+O813}, {\ttfamily L\+A\+T\+I\+N7}, {\ttfamily L\+A\+T\+I\+N-\/7}  &latin-\/7, Baltic rim character set  \\\cline{1-3}
{\ttfamily I\+S\+O-\/8859-\/14}  &{\ttfamily I\+S\+O885914}, {\ttfamily I\+S\+O8859-\/14}, {\ttfamily I\+S\+O-\/885914}, {\ttfamily I\+S\+O8859\+P14}, {\ttfamily I\+S\+O814}, {\ttfamily L\+A\+T\+I\+N8}, {\ttfamily L\+A\+T\+I\+N-\/8}  &latin-\/8, Celtic character set  \\\cline{1-3}
{\ttfamily I\+S\+O-\/8859-\/15}  &{\ttfamily I\+S\+O885915}, {\ttfamily I\+S\+O8859-\/15}, {\ttfamily I\+S\+O-\/885915}, {\ttfamily I\+S\+O8859\+P15}, {\ttfamily I\+S\+O815}, {\ttfamily L\+A\+T\+I\+N9}, {\ttfamily L\+A\+T\+I\+N-\/9}  &latin-\/9, Western European with euro symbol  \\\cline{1-3}
{\ttfamily I\+S\+O-\/8859-\/16}  &{\ttfamily I\+S\+O885916}, {\ttfamily I\+S\+O8859-\/16}, {\ttfamily I\+S\+O-\/885916}, {\ttfamily I\+S\+O8859\+P16}, {\ttfamily I\+S\+O816}, {\ttfamily L\+A\+T\+I\+N10}, {\ttfamily L\+A\+T\+I\+N-\/10}  &latin-\/10, Southeast European character set  \\\cline{1-3}
{\ttfamily K\+O\+I7}  &n/a &Russian\+: Kod Obmena Informatsiey, 7 bit characters  \\\cline{1-3}
{\ttfamily K\+O\+I8-\/R}  &{\ttfamily K\+O\+I8R}  &Russian\+: Kod Obmena Informatsiey, 8 bit  \\\cline{1-3}
{\ttfamily K\+O\+I8-\/U}  &{\ttfamily K\+O\+I8U}  &Ukrainian\+: Kod Obmena Informatsiey, 8 bit  \\\cline{1-3}
{\ttfamily U\+S-\/\+A\+S\+C\+II}  &{\ttfamily A\+S\+C\+II}, {\ttfamily U\+S\+A\+S\+C\+II}  &7-\/bit A\+S\+C\+II character set  \\\cline{1-3}
{\ttfamily U\+T\+F-\/8}  &{\ttfamily U\+T\+F8}  &variable-\/width universal character set  \\\cline{1-3}
{\ttfamily U\+T\+F-\/16}  &{\ttfamily U\+T\+F16}  &variable-\/width universal character set based on a fundamental 2-\/byte character encoding; not backwards-\/compatible with A\+S\+C\+II and therefore not supported universally in Qore; it\textquotesingle{}s recommended to convert these strings to U\+T\+F-\/8 in Qore; do not use U\+T\+F-\/16 as the default character encoding in Qore  \\\cline{1-3}
{\ttfamily U\+T\+F-\/16\+BE}  &{\ttfamily U\+T\+F16\+BE}  &variable-\/width universal character set based on a fundamental 2-\/byte character encoding with big-\/endian encoding; not backwards-\/compatible with A\+S\+C\+II and therefore not supported universally in Qore; it\textquotesingle{}s recommended to convert these strings to U\+T\+F-\/8 in Qore; do not use U\+T\+F-\/16\+BE as the default character encoding in Qore  \\\cline{1-3}
{\ttfamily U\+T\+F-\/16\+LE}  &{\ttfamily U\+T\+F16\+LE}  &variable-\/width universal character set based on a fundamental 2-\/byte character encoding with little-\/endian encoding; not backwards-\/compatible with A\+S\+C\+II and therefore not supported universally in Qore; it\textquotesingle{}s recommended to convert these strings to U\+T\+F-\/8 in Qore; do not use U\+T\+F-\/16\+LE as the default character encoding in Qore  \\\cline{1-3}
\end{longtabu}
\hypertarget{character_encoding_utf16_in_qore}{}\doxysubsection{U\+T\+F-\/16 Support in Qore}\label{character_encoding_utf16_in_qore}
U\+T\+F-\/16 is currently not well supported in Qore, because Qore\textquotesingle{}s string support is based on the assumption that all strings are backwards-\/compatible with A\+S\+C\+II, and U\+T\+F-\/16 is not due to the minimum 2-\/byte character width and the possibility of embedded null bytes.

It\textquotesingle{}s possible to generate string data in U\+T\+F-\/16 encoding (using \mbox{\hyperlink{group__string__functions_ga02c8ad6db7ae5c0930c8050840253035}{Qore\+::convert\+\_\+encoding()}}), however note that all strings so generated will be tagged with a B\+OM (byte order marker) at the beginning of the string data (this is performed by libiconv).

The following classes support parsing U\+T\+F-\/16 data by converting it to U\+T\+F-\/8 and processing the U\+T\+F-\/8 data\+:
\begin{DoxyItemize}
\item \mbox{\hyperlink{class_qore_1_1_data_line_iterator}{Qore\+::\+Data\+Line\+Iterator}}
\item \mbox{\hyperlink{class_qore_1_1_file_line_iterator}{Qore\+::\+File\+Line\+Iterator}}
\end{DoxyItemize}

The following classes support processing U\+T\+F-\/16 data natively\+:
\begin{DoxyItemize}
\item \mbox{\hyperlink{class_qore_1_1_buffered_stream_reader}{Qore\+::\+Buffered\+Stream\+Reader}}
\item \mbox{\hyperlink{class_qore_1_1_input_stream_line_iterator}{Qore\+::\+Input\+Stream\+Line\+Iterator}}
\item \mbox{\hyperlink{class_qore_1_1_stream_reader}{Qore\+::\+Stream\+Reader}}
\end{DoxyItemize}

Many string operations on U\+T\+F-\/16 data will provide invalid results due to the embedded nulls.

\begin{DoxyRefDesc}{Bug}
\item[\mbox{\hyperlink{bug__bug000001}{Bug}}]With the exception of the classes above that explicitly support U\+T\+F-\/16 data, B\+O\+Ms are ignored and all U\+T\+F-\/16 data is assumed to be big-\/endian; little-\/endian U\+T\+F-\/16-\/encoded data, even with a correct B\+OM, will not be processed correctly in Qore (in this case use the {\ttfamily U\+T\+F-\/16\+LE} encoding specifically)\end{DoxyRefDesc}
\hypertarget{character_encoding_default_encoding}{}\doxysection{Default Character Encoding}\label{character_encoding_default_encoding}
The default character encoding for Qore is determined by environment variables.

First, the {\ttfamily Q\+O\+R\+E\+\_\+\+C\+H\+A\+R\+S\+ET} environment variable is checked. If it is set, then this character encoding will be the default character encoding for the process. If not, then the {\ttfamily L\+A\+NG} environment variable is checked. If a character encoding is specified in the {\ttfamily L\+A\+NG} environment variable, then it will be used as the default character encoding. Otherwise, if no character encoding can be derived from the environment, {\ttfamily U\+T\+F-\/8} is assumed.

Character encodings are automatically converted by the Qore language when necessary. Encoding conversion errors will cause a Qore exception to be thrown. The character encoding conversions supported by Qore depend on the operating system\textquotesingle{}s {\ttfamily iconv} library function.

\begin{DoxyNote}{Note}
The \mbox{\hyperlink{group__misc__functions_ga8f800bec17c55a15ec7255dedfa25c6d}{get\+\_\+default\+\_\+encoding()}} function will return the default encoding for the Qore process.
\end{DoxyNote}
\hypertarget{character_encoding_encoding_examples}{}\doxysection{Character Encoding Usage Examples}\label{character_encoding_encoding_examples}
The following is a non-\/exhaustive list of examples in Qore where character encoding processing is performed.

Character encodings can be explicitly performed with the \mbox{\hyperlink{group__string__functions_ga02c8ad6db7ae5c0930c8050840253035}{convert\+\_\+encoding()}} function, and the encoding attached to a string can be checked with the \mbox{\hyperlink{group__string__functions_gac4a81651e374644e697e3ce61c373694}{get\+\_\+encoding()}} function. If you have a string with incorrect encoding and want to change the encoding tag of the string (without changing the actual bytes of the string), use the \mbox{\hyperlink{group__string__functions_gae583afd778a92a7b2a3027383fd75ed4}{force\+\_\+encoding()}} function.

\mbox{\hyperlink{group__misc__functions_ga8f800bec17c55a15ec7255dedfa25c6d}{get\+\_\+default\+\_\+encoding()}} returns the default encoding for the Qore process.

The \mbox{\hyperlink{class_qore_1_1_s_q_l_1_1_datasource}{Qore\+::\+S\+Q\+L\+::\+Datasource}}, \mbox{\hyperlink{class_qore_1_1_s_q_l_1_1_datasource_pool}{Qore\+::\+S\+Q\+L\+::\+Datasource\+Pool}}, and \mbox{\hyperlink{class_qore_1_1_s_q_l_1_1_s_q_l_statement}{Qore\+::\+S\+Q\+L\+::\+S\+Q\+L\+Statement}} classes will translate character encodings to the encoding required by the database if necessary as well (this is actually the responsibility of the D\+BI driver for the database in question).

The \mbox{\hyperlink{class_qore_1_1_file}{Qore\+::\+File}} and \mbox{\hyperlink{class_qore_1_1_socket}{Qore\+::\+Socket}} classes translate character encodings to the encoding specified for the object if necessary, as well as tagging strings received or read with the object\textquotesingle{}s encoding.

The \mbox{\hyperlink{class_qore_1_1_h_t_t_p_client}{Qore\+::\+H\+T\+T\+P\+Client}} class will translate character encodings to the encoding specified for the object if necessary, as well as tag strings received with the object\textquotesingle{}s encoding. Additionally, if an H\+T\+TP server response specifies a specific encoding to use, the encoding of strings read from the server will be automatically set to this encoding as well. 