How accurate is it?

By language

The language was recognised correctly in 98.3 % of the texts. Each language has only 4 to 5 test texts, so single percentages move a lot when one item is missed.

By type of data

Machine learning for names

Rules find names after greetings, titles and labels (“gospod Novak”, “Name: …”). Many names have no such cue: “Včeraj je Luka poklical”. For those, a small machine-learning model judges whether a capitalised word looks like a personal name, from its letters alone.

The model is a logistic regression on character patterns (171 KB). It was trained on lists of first names and surnames for Slovenian, Croatian, Serbian, Bosnian, German, English, Danish, Norwegian, Swedish, Finnish and Icelandic, against the most frequent ordinary words of those languages and a list of place names. On names it never saw during training it separates names from other words with an area under the ROC curve of 0.96. It runs inside the service and in the browser extension; no text is sent anywhere. Processing a test text, including the model, took 0.7 ms on average.

It is stricter at the start of a sentence and in German, where every noun is capitalised. A word that also appears in lowercase in the same text is never treated as a name, and phrases such as “Luka Koper” or “Deutsche Telekom”, where one word is clearly not a name, are left alone. You can switch it off with the “Names without cues” option.

How we measure

• Test texts: 60 realistic e-mails, notes, forms and chat messages in Slovenian, Croatian, Serbian (Latin and Cyrillic), Bosnian, German (Germany, Austria, Switzerland), English, Danish, Norwegian, Swedish, Finnish and Icelandic, including 7 texts without any personal data. All people and numbers are invented; ID numbers, IBANs and card numbers have valid check digits, as real ones do.
• Marking: every name, address, postcode with town, phone number, e-mail, ID and tax number, bank and card number, date of birth, customer or document number, password and IP address was marked by hand. Company names and cities mentioned in passing are not personal data and are not marked; replacing them counts as a false alarm.
• Replaced: share of marked items whose every letter and digit was replaced. A name with only the first name replaced counts as a miss.
• Precision: share of replacements that overlap a marked item.
• Separate development set: rules and thresholds were adjusted on 46 other texts. For the seven original languages the test texts were never used for tuning. For Danish, Norwegian, Swedish, Finnish and Icelandic they were: the first measurement is published in the history below, and the rules were then corrected against the items those texts missed.
• Language: detected automatically, as in normal use.

The test texts and the scoring program are part of the source code (bench/ and bin/bench.php), so the numbers can be reproduced with php bin/bench.php.

What it does not catch

• Single first names without any cue are sometimes missed, especially in German and English, and at the start of a sentence.
• Finnish and Icelandic names in a case form whose dictionary form does not appear anywhere in the same text.
• A Danish CPR number issued after 2007 need not satisfy the old check digit, so one written without its hyphen and without a label may be missed.
• Nicknames, initials (“J. N.”), job titles and other indirect details that identify a person (“the CEO of our Koper branch”).
• Numbers without a label and without a check digit, for example most customer numbers written on their own.
• Text inside images and scanned documents.
• US state codes and parts of UK addresses are kept on purpose; they rarely identify a person alone.

Always read the result before you share it. For confidential data, run the software on your own server (see self-hosting).

History