|
The main aim of this page is to determine
A GENERALLY ACCEPTED SORT ORDER
for
Unicode Tamil characters
Edition 1 (Through out this document, the word 'Tamil' has to be pronounced as
' ' / 'thamizh' )
This document deals with the topics like,
I started to write this document, when I tried to developed an algorithm to compare two
Tamil Unicode strings. The corresponding C Code is here. It basically
uses an array of sorted Tamil Unicode positions and a comparison logic that uses this array
to justify the two Unicode Tamil strings to be compared.
|
Problems in alphabetic sorting with UNICODE Tamil Encoding
Following (Table 1) is table that contains all the possible Tamil characters that are
supported by The Unicode Standard,
Version 4.0 in a sorted maner. The original position of
Tamil in Unicode chart starts
from B80 and extends up to BFF, a total of 127 allocations.
If you look at the Chart for Tamil
in Unicode, the order in which the alphabets are arranged are not at all in the
conventional, natural, traditional and standard way,
that modern Tamil uses. This can be considered as the primary issue.
Since Unicode is not in a position to rearrange the
current Tamil Unicode scheme, It is wise for us, atleast to standardize the Unicode Tamil
ordering/sorting scheme to our best.
The main flaws in Unicode Tamil ordering are:
Flaw 1: ' ' (aaytham - B83) comes before vowels.
Soln: Sorting algorithm should arrange this after
![](../../padanGaL/tam_o.gif) (au - B94).
Flaw 2: Vowel ![](../../padanGaL/tam_o.gif)
(au - B94) has been given the flexibility of representing it as
two different letters as (o - B92) +
'![](../../padanGaL/tam_x.gif) ' ( `au - BD7). This
flexibility creates lot of problems by having two different character sequences for a same
word that includes the letter
(au).
Soln: A Standard has to be devised to eliminate the usage of
'![](../../padanGaL/tam_x.gif) ' (`au - BD7).
Flaw 3: Similarly, Vowel modifiers for ; (o - B92),
(O - B93) and
(au - B94) such as '
![](../../padanGaL/tam_x.gif) '
( `o - BCA), '![](../../padanGaL/tam_xee.gif)
' ( `O - BCB) and
'![](../../padanGaL/tam_xee.gif)
'
( `au - BCC) are also given the flexibility as follows
- "consonant +
![](../../padanGaL/tam_x.gif)
( `o - BCA)" can aslo be represented as
"
( `e - BC6) + consonant +
![](../../padanGaL/tam_x.gif) ( `aa - BBE)"
- "consonant +
![](../../padanGaL/tam_x.gif) ( `O - BCB)" can aslo be represented as
"
( `E - BC7) + consonant +
![](../../padanGaL/tam_x.gif) ( `aa - BBE)"
- "consonant +ௌ ( `au - BCC)" can aslo be represented as
"
( `e - BC6) + consonant +
![](../../padanGaL/tam_x.gif) ( `au - BD7)"
This flexibility creates lot of problems by having
two different character sequences for a same word that includes the letter ஒ, ஓ
or ஔ.
Soln: A Standard has to be devised to eliminate the usage of
- "
( `e - BC6) + consonant +
![](../../padanGaL/tam_x.gif) ( `aa - BBE)",
- "
( `E - BC7) + consonant +
![](../../padanGaL/tam_x.gif) ( `aa - BBE)" and
- "
( `e - BC6) + consonant +
![](../../padanGaL/tam_x.gif) ( `au - BD7)"
Flaw 4: (ja - B9C) comes in between
(cha - B9A) and
(nYa - B9E) as
in any other Non-Tamil languages and Gerantha Script.
Soln: Sorting algorithm should arrange this as a separate set of characters after the
regular Tamil consonants ( ).
Flaw 5: Positions of (na - BA9),
(Ra - BB1),
(La - BB3) and (zha - BB4) are totally wrong.
Soln: Sorting algorithm should take care of it.
Flaw 6: Order of Tamil Grantha letters (ja - B9C),
(Ca - BB6) [Not 'cha' - B9A],
(sha - BB7), (Sa - BB8), (ha - BB9),
(ksha - B95+BCD+BB7) and
(sree - BB8+BCD+BB0+BC0) are not even internationally
standardized.
Soln: We need to find a unique sort order for these Tamil Grantha letters and the sorting
algorithm should take care of this ordering.
Flaw 7: Tamil digits are positioned after alphabets and vowel modifier symbols. This is
the most unconventional ordering. Tamil digits should be ordered before
vowels.
Soln: Sorting algorithm should take care of this ordering.
Flaw 8: Normally (Rs - BF9) is used as a currency
symbol in front of numbers (eg. .5000). So, this symbol
should be ordered before Tamil digits.
Soln: Sorting algorithm should take care of this ordering.
Flaw 9: Normally (`# - BFA) is used as a number
symbol (same as the usage of '#') in front of numbers (eg. 5).
So, this symbol should at least be ordered before Tamil digits or to be considered having equal priority with
'#' (hash - 0x23).
Soln: We should decide upon this and sorting algorithm should take care of this ordering.
Flaw 10: Tamil digit O (zero - BE6), even though is not available in the conventional
way of representing Tamil numbers, has to be included
in Unicode ordering mechanism to cope with international number representations.
Soln: Proposal documents are there
in internet that requests the inclusion of zero (depicted as 'O'), along with other Tamil
digits in Unicode Tamil. But the
document supplied by unicode.org does not have this
in the chart.
Flaw 11 (less important): The very special combinatory letter
' ' (fa) is
being widely used in names (such as fizal, fathima). But it fall in the category of
'most non standard alphabets of Tamil'. The position in the ordering of this
letter can be considered just after the Tamil Grantha list.
Soln: We should decide upon this and sorting algorithm should take care of this ordering.
Flaw 12 (less important): Similarly the (Ca - BB6)
[Not 'cha' - B9A] has to be listed along with the Tamil Grantha
alphabets, as it is very much used in spiritual Tamil scripts.
Soln: We need to initiate Unicode proposals regarding this.
|
^ |
![](../../padanGaL/tamil_sort_order.gif) (Table 1) |
^ |
I request you to go through the above table. In the above table, each number in the first column
represent a set of individual traditional sorted alphabet blocks. They are,
- Rupees Symbol
- Number Symbol
- Tamil Digits
- Tamil Vowels
- Primary Tamil Consonants
- Sa Series
- Sha Series
- Ja Series
- Ha Series
- Ksha Series
- Ca [Not 'cha'] Series
- Sree
- Tamil Vowel Modifiers
- Day Symbol
- Month Symbol
- Year Symbol
- Credit Symbol
- Debit Symbol
- Ditto Symbol
I hope, the above order is correct according to me.
I request you to review this and suggest me the corrections that are to be made in the above list of
sortings. For Example, If you feel the order of Tamil Grantha Series are not correct, and you want
the new order to be like
Ja, Sa, Sha, Ha, Ksha, Sree and Ca [Not 'cha']
Then, please let me know the entire new order (with respect to the current order specified above) as follows.
1, 2, 3, 4, 5, 8, 6, 7, 9, 10, 12, 11, 13, 14, 15, 16, 17, 18, 19
Enter Your
Suggestions
Here
The above link is a Guest book where you can file your suggestions. I will be updating this
page with all your valuable suggestions. See previous suggestions in guest book
You can also post thru Group or
mail to tamilsortorder@yahoo.com
|
^ |
Other major Flaws with UNICODE Tamil Encoding
Apart from the above-specified Tamil Unicode ordering problem, there are lot
of grammatical and syntactical flaws
in Tamil Unicode. They include
-
Consonants represented in Unicode, should be perfect Mey. That is, they should be
represented as (
...) and should not be like
what is now ( ...). The current
representation violates the grammar. The actual conventional, correct and natural
Tamil grammar is as follows.
(k) + ` ( `a)
becomes (ka)
(ch) +
![](../../padanGaL/tam_x.gif) ( `u)
becomes (chu)
But according to Unicode coding scheme, the entire grammar is being reversed as follows.
(ka) +
(pulli) becomes
(k)
(cha) +
![](../../padanGaL/tam_x.gif) ( `u)
becomes (chu)
(ta) +
![](../../padanGaL/tam_xee.gif)
( `O) becomes
![](../../padanGaL/tam_xee.gif)
(tO)
This is totally wrong and a BIG blunder. This is also the most un-natural way
of representing Tamil language alphabets. This will create lot of problems in future
in areas such as,
- Tamil Natural language processing
- Tamil Database processing
- Tamil Computer language development
' ' (aaytham - B83) is represented as a vowel modifier in Unicode Tamil.
It should be represented as an individual alphabet.
|
^ |
Notes
From Various documents and newsgroups, I came to know that, the current
scheme of Tamil in Unicode can NEVER be changed. The only possibility
according to Unicode is that we can add some Tamil symbols that fill the free gaps, that too
with greater pain.
The Funny and shameful thing here is, "Department of Information Technology, Government
of Tamil Nadu and Tamil Virtual University" and "Ministry of Information Technology,
Government of India" are Full Corporate Members of
Unicode Consortium. I also
came to know that, these representatives of Indian/Tamil languages have raised very less
issues regarding the above-discussed things, in many of the Unicode Consortium meetings
and discussions. Actually we Tamil people are the most unlucky to have such a irresponsible
Governments, Universities and Research Institutes that are spread all over the world. India
(having Tamil as one among the 18 state languages), Singapore (having Tamil as one of the
national language), Sri Lanka (having Tamil as one of the national language), USA and
other places.
Regarding the ordering scheme in Unicode Tamil and the most un-natural way of representing
alphabets and combinatory alphabets, Unicode has the following
points to tell.
- The unicode ordering scheme has nothing to do with the natural and traditional ordering scheme
of any language.
- It is the resposibility of the algorithm that uses unicode, to fix the sort order.
- It is well accepted that very few languages that have Unicode standard enjoy natural and
traditional ordering.
- As lot of softwares and documents already implemented the current system of Unicode
standard. So, It is very hard to introduce any change in the sort order in Unicode.
Organizations like Microsoft (a Full Corporate Member of Unicode) are expressing
complete opposition for change in unicode. (Organizations that promote Tamil
computing have given very less supported the change in Unicode)
- Unicode has adopted the ISCII coding scheme devoleped and recomended by Department of
Electronics and Indian Government (During this recomendation, Tamil Scholars are
not cosulted.)
themselves
agree to an extend by saying that,
Somebody are even trying to propose new set of slots in Unicode for the entire set of Tamil alphabets
and symbols in a perfect manner. But I don't know how far they will succeed. If it is done, we are
the luckiest.
Although the above topics are discussed lots and lots of times at many news groups, meetings
and conferences, there are very less resources on the internet to describe the standard sorting
order scheme for Tamil.
I heard that Microsoft has supplied the API CompareString, which is capable of two Tamil
Strings. Are there any specification about "which ordering scheme it is using"?
References:
|
^ |
Please post your comments and suggestions through any of the following means:
Suggestions
Author: R.Padmakumar
Date: 20-Dec-2004
![](http://counter.digits.com/wc/-d/4/thamizh_unicode_sort)
| | |