One script. Three languages.
Finally told apart.
Sinhala, Pali and Sanskrit share the same letters and often the same page. Today's language identifiers see Sinhala script and answer “Sinhala”. LangID tells you, sentence by sentence, which language you're actually reading, in pasted text or scanned PDFs.
- F1 on Sinhala, Pali & Sanskrit
- 99.9%
- F1 on Sinhala, Pali & Sanskrit
- Pali & Sanskrit, vs. off-the-shelf
- 0 → 99.9%
- Pali & Sanskrit, vs. off-the-shelf
- other languages still recognised by GlotLID
- 2,100+
- other languages still recognised by GlotLID
- models to choose from
- 6
- models to choose from
The gap
For a reader, it's natural. For a computer, it's a puzzle.
For more than two thousand years one script has carried three languages: Sinhala, the living language of Sri Lanka; Pali, the language of the Buddha's teachings; and Sanskrit, the classical language of learning. A Sinhala commentary quotes a Pali verse, then a line of Sanskrit, all in the same letters.
Same script. Same page. Three languages.
ධම්මපදය බෞද්ධ සාහිත්යයේ වඩාත් ප්රසිද්ධ ග්රන්ථයකි.
මනොපුබ්බඞ්ගමා ධම්මා මනොසෙට්ඨා මනොමයා.
මාමකාඃ පාණ්ඩවාශ්චෛව කිමකුර්වත සඤ්ජය.
ධම්මපදය බෞද්ධ සාහිත්යයේ වඩාත් ප්රසිද්ධ ග්රන්ථයකි.Sinhala
මනොපුබ්බඞ්ගමා ධම්මා මනොසෙට්ඨා මනොමයා.Pali
මාමකාඃ පාණ්ඩවාශ්චෛව කිමකුර්වත සඤ්ජය.Sanskrit
F1 score on a held-out test set (GlotLID v3)
Sinhala
Pali
Sanskrit
Every leading open language identifier we tested scored zero on Pali and Sanskrit: it labels them all as Sinhala. LangID's fine-tuned models get them right.
Corpora stay contaminated
Web-scale datasets filter by language ID. Pali and Sanskrit in Sinhala script quietly end up labelled as Sinhala, polluting all three.
Archives can't be searched
Digitised books, journals and transcribed palm-leaf manuscripts mix the three languages. Without labels, nobody can find the verses inside.
Scholars work by hand
Knowing where the Pali starts and the Sinhala ends still means reading every line. LangID does the first pass in seconds.
How it works
Three steps from page to languages
No setup, no code. Bring a passage or a whole book.
- 1
Bring your text
Paste a passage, or upload a scanned PDF. OCR reads Sinhala, Sanskrit and English pages for you.
- 2
Every sentence identified
LangID splits the text and identifies the language of each sentence, with a confidence score.
- 3
Read, correct, export
Explore colour-coded results, fix anything that's wrong, and export everything as CSV.
Product tour
See what's really on the page
Real screens from LangID, running on a real mixed-language passage.
Classify text
Paste any Sinhala-script text, pick a model
Choose from six models: a fast three-way baseline, and five leading open language identifiers (NLLB, GlotLID, OpenLID, ConLID and fastText), each fine-tuned on Sinhala, Pali and Sanskrit.
- Trained with rehearsal, so they still recognise hundreds of other languages (over 2,000 with GlotLID)
- Split by sentence, paragraph, full text, or automatically
- Long passages stream back batch by batch
Read the results
Every language at a glance
Your text keeps its original layout, and every sentence is highlighted in its language's colour: blue for Sinhala, green for Pali, violet for Sanskrit. Click a language to fade everything else out.
- Distribution bar shows how much of the text is in each language
- Focus on one language to find every Pali verse instantly
- English, Tamil, Hindi and others get colours of their own
Confidence
Know exactly how sure the model is
Hover any sentence to see its confidence and how the probability is spread across candidate languages. Uncertain sentences get a dashed underline, so you know where to look twice.
- Per-sentence confidence score
- Top candidate languages with probabilities
- Export sentence, language and confidence as CSV
Documents
From scanned page to languages
Upload a scanned PDF and LangID reads it page by page with OCR, then identifies the language of every sentence on every page. Results appear as each page finishes, so even whole books are never a long wait.
- Tesseract for speed, or Surya's transformer OCR for harder scans
- Original page beside the classified text; two-column pages kept in reading order
- Cancel anytime: finished pages are kept and the rest is refunded
Better with every correction
Spot a mistake? Teach the model.
No model is perfect. Click any sentence, choose the right language and submit. Corrections go to a review queue and, once approved, become training data for the next version of the models.
- Report → review → approve → retrain
- Admins export the approved set for retraining
- The more LangID is used, the better it gets
Built on research
Near-perfect, without forgetting the rest of the world
LangID grew out of a dedicated study of Sinhala-script language identification. Each model was fine-tuned on a carefully built, leakage-free dataset of Sinhala, Pali and Sanskrit, and trained with rehearsal so it keeps recognising the languages it already knew.
99.96%
best macro-F1 on Sinhala, Pali & Sanskrit
≥ 99.4%
macro-F1 on the FLORES+ benchmark, after fine-tuning
F1 per language, held-out test set
Fine-tuned with rehearsal
| Model | Sinhala | Pali | Sanskrit | Macro |
|---|---|---|---|---|
| OpenLID v3 | 99.94% | 99.92% | 100% | 99.96% |
| NLLB LID-218 | 99.95% | 99.94% | 99.96% | 99.95% |
| GlotLID v3 | 99.93% | 99.92% | 99.92% | 99.92% |
| fastText LID-176 | 99.88% | 99.82% | 99.66% | 99.79% |
| ConLID | 99.33% | 99.07% | 96.93% | 98.44% |
The same models off the shelf: about 72% on Sinhala, 0% on Pali and Sanskrit.
Who it's for
Built for the people who work with these texts
Scholars & students
Of Buddhist literature, Pali and Sanskrit: find every verse in a commentary.
Monastic libraries
Catalogue collections by language without reading every page.
Archives & digitisation
Turn scanned books and journals into language-labelled text.
NLP researchers
Build clean Sinhala, Pali and Sanskrit corpora, sentence by sentence.
Simple pricing
Pay only for what you use
Top up your balance once and spend it as you go. OCR is charged per page and language identification per word, and every charge and refund is listed on your Usage page.
- Top up in Sri Lankan rupees via PayHere: card, mobile wallet or online banking
- Cancelled work is refunded automatically
- Your dashboard keeps all your texts and documents one click away
