Queen and Country

Can Sonnet detect the difference between en_US and en_GB? No, and yes. I've been asked that quite a bit, so I'll clarify. There are several key requirements, as I see them, for adding language detection to KDE.

1. It must distinguish between different languages. While all supported languages could be detected, it is more likely that a user will only use a few languages in most of his sessions or that the program will be used by in setting by people speaking a specific range of languages. For example, a speaker of French and German, or school computer lab with speakers of Xhosa, Zulu and Afrikaans.
2. It must be Fast. The detection must occur in real-time; otherwise, the user might as well select their language manually.
   

I've optimized the language detection for the above criteria. The models used are limited so that they are fast. This makes detection less accurate, but this is overcome, by taking into account other factors such default settings or the language detected for surrounding text segments.

Are users of multiple dialects less important? No, but if I value priorities (1) and (2) above, then the detection of the dialect will be highly inaccurate due to the nature of the algorithms used. However, detecting a sample of text written in a dialect as the common language currently is robust. It must be, the detection would be useless for our purposes if it were so brittle as to fail on minor spelling differences, which are likely candidates for errors. Additionally the statistical differences between dialects are much less then between different languages. In most cases, text in a different dialect just isn't contrastive enough in the greater scheme of things.

Now, there are ways to distinguish between American and British spellings, but doing it at the statistical comparison stage isn't reliable with the current heuristics used, but could be done if there was enough user demand. However, any application that utilizes the languages guessing class should be a bit smarter. The convenience classes Sonnet will provide will distinguish between countries by checking user settings. If the users locale is de_BE and French is detected then the spell checker would default to checking with fr_BE; it would fallback to fr or fr_FR if a dictionary for fr_BE was not found.

In summary, the language detection class will not distinguish between cases where the same language has a different orthography in a different country. Yet, tools built using this class will distinguish between them using alternative means. But, I could be wrong in my assumptions. If enough complaints surface, it's early enough to change the behavior.

For all those who emailed me questions similar to ris:

[08:28] <ris> rideout: re: sonnet test sentences - would it be useful to put a british english test sentence in? would sonnet be able to distinguish between the two if you stuck a few 'realise', 'centre' and 'programme's in?

It is very reasonable to test that British English is detected as English. Empirically, Sonnet is not encumbered by the mild differences that exist between British and American orthography.

EDIT: I meant to ask this originally, but became carried away and forgot. Are there languages for which I must distinguish between dialects? I can think of examples like Chinese where the dialects are essentially different languages. But, in this case they all share a common orthography. Are there those with different orthographies?

NOTE: In this post, I did conflate the notion of dialects and countries with different standards of orthography. To the pedantic, please don't shoot. To the curious, I try to limit the use of technical linguistic terms in this blog and to use the imprecise and ofter confused vernacular, this has led to some confusion in the past.