<!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook V4.4//EN"
 "http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd"
[
     <!ENTITY % local SYSTEM "local.ent">
     %local;
     <!ENTITY % entities SYSTEM "entities.ent">
     %entities;
     <!ENTITY % idcommon SYSTEM "common/common.ent">
     %idcommon;
]>
<refentry id="yaz-icu">
 <refentryinfo>
  <productname>YAZ</productname>
  <productnumber>&version;</productnumber>
  <orgname>Index Data</orgname>
 </refentryinfo>

 <refmeta>
  <refentrytitle>yaz-icu</refentrytitle>
  <manvolnum>1</manvolnum>
  <refmiscinfo class="manual">Commands</refmiscinfo>
 </refmeta>

 <refnamediv>
  <refname>yaz-icu</refname>
  <refpurpose>YAZ ICU utility</refpurpose>
 </refnamediv>

 <refsynopsisdiv>
  <cmdsynopsis>
   <command>yaz-icu</command>
   <arg>-c <replaceable>config</replaceable></arg>
   <arg>-p <replaceable>opt</replaceable></arg>
   <arg>-s</arg>
   <arg>-x</arg>
   <arg choice="opt">infile</arg>
  </cmdsynopsis>
 </refsynopsisdiv>

 <refsect1><title>DESCRIPTION</title>
  <para>
   <command>yaz-icu</command> is a utility which demonstrates
   the ICU chain module of yaz. (<filename>yaz/icu.h</filename>).
  </para>
  <para>
    The utility can be used in two ways. It may read some text
    using an XML configuration for configuring ICU and show text analysis.
    This mode is triggered by option <literal>-c</literal> which specifies
    the configuration to be used. The input file is read from standard
    input or from a file if <literal>infile</literal> is specified.
  </para>
  <para>
    The utility may also show ICU information. This is triggered by
    option <literal>-p</literal>.
  </para>
 </refsect1>

 <refsect1><title>OPTIONS</title>
  <variablelist>
   <varlistentry>
    <term>-c <replaceable>config</replaceable></term>
    <listitem><para>
      Specifies the file containing ICU chain configuration
      which is XML based.
     </para></listitem>
   </varlistentry>

   <varlistentry>
    <term>-p <replaceable>type</replaceable></term>
    <listitem><para>
      Specifies extra information to be printed about the ICU system.
      If <replaceable>type</replaceable> is <literal>c</literal>
      then ICU converters are printed.
      If <replaceable>type</replaceable> is <literal>l</literal>,
      then available locales are printed.
      If <replaceable>type</replaceable> is <literal>t</literal>,
      then available transliterators are printed.
     </para></listitem>
   </varlistentry>

   <varlistentry>
    <term>-s</term>
    <listitem><para>
      Specifies that output should include sort key as well. Note that
      sort key differs between ICU versions.
     </para></listitem>
   </varlistentry>

   <varlistentry>
    <term>-x</term>
    <listitem><para>
      Specifies that output should be XML based rather than
      "text" based.
     </para></listitem>
   </varlistentry>

  </variablelist>
 </refsect1>
 <refsect1><title>ICU chain configuration</title>
  <para>
   The ICU chain configuration specifies one or more rules to convert
   text data into tokens. The configuration format is XML based.
  </para>
  <para>
   The toplevel element must be named <literal>icu_chain</literal>.
   The <literal>icu_chain</literal> element has one required attribute
   <literal>locale</literal> which specifies the ICU locale to be used
   in the conversion steps.
  </para>
  <para>
   The <literal>icu_chain</literal> element must include elements where
   each element specifies a conversion step. The conversion is performed
   in the order in which the conversion steps are specified.
   Each conversion element takes one attribute: <literal>rule</literal>
   which serves as argument to the conversion step.
  </para>
  <para>
   The following conversion elements are available:

   <variablelist>
    <varlistentry>
     <term>casemap</term>
     <listitem><para>
       Converts case (and rule specifies how):

       <variablelist>
	<varlistentry>
	 <term>l</term>
	 <listitem>
	  <para>Lower case using ICU function u_strToLower. </para>
	 </listitem>
	</varlistentry>

	<varlistentry>
	 <term>u</term>
	 <listitem>
	  <para>Upper case using ICU function u_strToUpper.</para>
	 </listitem>
	</varlistentry>

	<varlistentry>
	 <term>t</term>
	 <listitem>
	  <para>To title using ICU function u_strToTitle.</para>
	 </listitem>
	</varlistentry>

	<varlistentry>
	 <term>f</term>
	 <listitem>
	  <para>Fold case using ICU function u_strFoldCase.</para>
	 </listitem>
	</varlistentry>

       </variablelist>
      </para></listitem>
    </varlistentry>

    <varlistentry>
     <term>display</term>
     <listitem><para>
       This is a meta step which specifies that a term/token is to
       be displayed. This term is retrieved in an application
       using function icu_chain_token_display (<filename>yaz/icu.h</filename>).
      </para></listitem>
    </varlistentry>

    <varlistentry>
     <term>transform</term>
     <listitem><para>
       Specifies an ICU transform rule using a transliterator
       Identifier.
       The rule attribute is the transliterator Identifier.
       See  <ulink url="&url.icu.transform;">ICU Transforms</ulink> for
       more information.
      </para></listitem>
    </varlistentry>

    <varlistentry>
     <term>transliterate</term>
     <listitem><para>
       Specifies a rule-based transliterator.
       The rule attribute is the custom transformation rule to be used.
       See <ulink url="&url.icu.transform;">ICU Transforms</ulink> for
       more information.
      </para></listitem>
    </varlistentry>

    <varlistentry>
     <term>tokenize</term>
     <listitem><para>
       Breaks / tokenizes a string into components using
       ICU functions ubrk_open, ubrk_setText, .. . The rule is
       one of:
       <variablelist>
	<varlistentry>
	 <term>l</term>
	 <listitem>
	  <para>Line. ICU: UBRK_LINE.</para>
	 </listitem>
	</varlistentry>

	<varlistentry>
	 <term>s</term>
	 <listitem>
	  <para>Sentence. ICU: UBRK_SENTENCE.</para>
	 </listitem>
	</varlistentry>

	<varlistentry>
	 <term>w</term>
	 <listitem>
	  <para>Word. ICU: UBRK_WORD.</para>
	 </listitem>
	</varlistentry>

	<varlistentry>
	 <term>c</term>
	 <listitem>
	  <para>Character. ICU: UBRK_CHARACTER.</para>
	 </listitem>
	</varlistentry>

	<varlistentry>
	 <term>t</term>
	 <listitem>
	  <para>Title. ICU: UBRK_TITLE.</para>
	 </listitem>
	</varlistentry>

       </variablelist>
      </para></listitem>
    </varlistentry>

    <varlistentry>
     <term>join</term>
     <listitem>
      <para>
       Joins tokens into one string. The rule attribute is the joining
       string, which may be empty. The join conversion element was added
       in YAZ 4.2.49.
      </para>
     </listitem>
    </varlistentry>
   </variablelist>

  </para>
 </refsect1>
 <refsect1><title>EXAMPLES</title>
  <para>
   The following command analyzes text in file <filename>text</filename>
   using ICU chain configuration <filename>chain.xml</filename>:
   <screen>
    cat text | yaz-icu -c chain.xml
   </screen>
   The chain.xml might look as follows:
    <screen><![CDATA[
<icu_chain locale="en">
  <transform rule="[:Control:] Any-Remove"/>
  <tokenize rule="w"/>
  <transform rule="[[:WhiteSpace:][:Punctuation:]] Remove"/>
  <transliterate rule="xy > z;"/>
  <display/>
  <casemap rule="l"/>
</icu_chain>
]]>
   </screen>
  </para>
 </refsect1>
 <refsect1><title>SEE ALSO</title>
  <para>
   <citerefentry>
    <refentrytitle>yaz</refentrytitle>
    <manvolnum>7</manvolnum>
   </citerefentry>
  </para>
  <para>
   <ulink url="&url.icu;">ICU Home</ulink>
  </para>
  <para>
   <ulink url="&url.icu.transform;">ICU Transforms</ulink>
  </para>
 </refsect1>
</refentry>

<!-- Keep this comment at the end of the file
Local variables:
mode: nxml
nxml-child-indent: 1
End:
-->
