Writing a dictionary software for a spoken language lazarus / 21-General ---------------------------------------------------------------------- bobby100 · 2024-09-01 15:46 Dear forum members, I am a speaker of a language that will probably get extinct with my generation. It is a variation of Romanian language spoken in Serbia. It is a branch that didn't evolve together with Romanian for last 100 years. It diverges a lot from standard Romanian etc. etc. It is more like medieval Romanian. I would like to get it written down, and here I seek for advices about making a proper software for such task. One word could have the following attributes: - descriptive writing of the word capable of showing accents, ligatures (underscore the letters), silent letters (striked through) etc. - type (verb, adjective etc.) - finite group of options - description - text of an unknown length - usage examples - list of strings - audio file(s) - links to files on HDD - graphic/video file(s) - links to files on HDD - synonym(s) - strings that can also be searched for in the database - translation to other languages - should be possible to get it indexed per language, if a foreigner searches for translation - grammar (this is a whole category of tables) I understand that the databases are the best way to store data that can be quickly searched by an end-user, but this is where I fail. I do know how to work with databases for some simple things, but I struggle here with unknown number of fields and unknown length of the fields. For example, I can't predict how many synonyms will one word have, or how many examples will be there for one word, and also how much text will that be. Could some, please, give me some hints how could I organize all this? ---------------------------------------------------------------------- MarkMLl · 2024-09-01 15:59 I can't speak for lesser databases, but my experience with PostgeSQL is that it is entirely happy with variable-length strings, is entirely happy with adding columns (e.g. with an initial nil value), entirely happy with attached binaries of modest size, and has good character support provided that you can set this for (at least) a table and (possibly- would have to check) the entire database. If you started off with that and used it while you were massaging your storage and organisation into shape you could possibly later downgrade to e.g. SQLite. Having said that, it would not surprise me at all if there were existing tools to do this... possibly proprietary to a particular university etc. but for something important like you're suggesting they might offer some /very/ attractive deals. MarkMLl ---------------------------------------------------------------------- Aruna · 2024-09-01 16:18 This may be useful ---------------------------------------------------------------------- BrunoK · 2024-09-01 17:13 Go sqlite3 to start. ---------------------------------------------------------------------- bobby100 · 2024-09-01 19:11 I'll take a look at the existing programs, but all I've found till now are made for English-speaking researchers that are researching foreign languages. ---------------------------------------------------------------------- alpine · 2024-09-01 20:19 My native language is one from the South Slavic group so I deeply sympathize. Quote from: bobby100 on September 01, 2024, 05:46:44 pm*snip* I understand that the databases are the best way to store data that can be quickly searched by an end-user, but this is where I fail. I do know how to work with databases for some simple things, but I struggle here with unknown number of fields and unknown length of the fields. For example, I can't predict how many synonyms will one word have, or how many examples will be there for one word, and also how much text will that be. Could some, please, give me some hints how could I organize all this? In addition to what Mark said about PostgreSQL, I will note that sqlite also has similar characteristics. It is quite happy with columns with variable width and you can put structured text (XML, JSON) in one column instead of making endless tail continuations, etc. Also, relatively small blobs can be stored easily encoded with base64. Of course it depends on purpose as PostgreSQL and sqlite are quite different in other ways, the former being full blown server and the latter - an embedded local library with SQL queries support.  As for the database model, I guess people will be found to help (I would) but it is very important to do a good research at the (natural) language level what the entities and relationships are. Are there any existing organizations and/or existing studies? After all, we're CSc specialists, not linguists, right. Quote- descriptive writing of the word capable of showing accents, ligatures (underscore the letters), silent letters (striked through) etc.For my language OS support has never been 100% adequate and over the years I've always had sometimes intractable issues. And it is nothing special - no accents nor ligatures, just the basic Cyrillic letters. I don't have any idea how your language is written, but the question is whether there is adequate typographical support in modern OS-es or you should implement some sort of enhanced graphic tools to support it? ---------------------------------------------------------------------- Sieben · 2024-09-01 21:40 Quote from: bobby100 on September 01, 2024, 05:46:44 pm- descriptive writing of the word capable of showing accents, ligatures (underscore the letters), silent letters (striked through) etc. As for the phonetic representation of a word - the pronounciation - you should definitely have a look at the International Phonetic Alphabet: https://en.wikipedia.org/wiki/International_Phonetic_Alphabet although you might have to consult a linguist which might be a good idea anyway. After all, a complex task like this might even require a team of sorts. As for databases - no offense intended, I could be completely wrong - your remarks sound a bit like you are viewing a database more or less as a single table. You might have to take a deeper look at the concept of Relational Databases. That said, SQLite indeed seems to be a reasonable candidate. On how to organize all this - I'd reach out to the major universities of your country and ask if they can name anyone who is working on that language and then ask those if they are aware of any such projects or if they are interested in creating one. ---------------------------------------------------------------------- egsuh · 2024-09-02 04:11 QuoteI do know how to work with databases for some simple things, but I struggle here with unknown number of fields and unknown length of the fields. For example, I can't predict how many synonyms will one word have, or how many examples will be there for one word, and also how much text will that be. These are not very difficult issues, but they depend on DB design, not which database you use. Some issues are simple. For example, - Unknown length of the fields :    BLOB type files simply solve this. - unknown number of fields :  Two approaches (at least) are possible    1)  with BLOB field, you may store multiple lines of text. In Lazarus, you can import the contents into TStrings.    2)  Using separate table, with one record  for each item (for example, a synonym). And you have to solve the issue programmatically.   But if you are doing something with Lazarus (or other Language), "unknown number of fields" is not a serious issue. Problem lies when one item is not a single line. But there are not a big deal if treated through Lazarus, etc. Basically your issues are DB design issues, I think. Or you may use non-SQL DBs, like MongoDB. I personally don't think they have large merit over SQL DBs. ---------------------------------------------------------------------- paweld · 2024-09-02 05:52 I would create more or less such a database structure:Code: [Select]table "types": - "typeId" - int autoincrement - "name" - varchar(50): verb, adjective etc. table "words": - "wordId" - int autoincrement - "word" - text - "???" - text - descriptive writing of the word capable of showing accents, ligatures (underscore the letters), silent letters (striked through) etc. - "typeId" - int - from "types" table - "description" - text table "samples": - "sampleId" - int autoincrement - "wordId" - int - from "words" table - "sample" - text table "attachments": - "attachmentId" - int autoincrement - "wordId" - int - from "words" table - "type" - char(1) - A-audio, G-graphic, V-video - "filename" - text - with path table "synonims": - "wordId" - int - from "words" table - "synonimWordId" - int - from "words" table table "translations": - "translateId" - int autoincrement - "wordId" - int - from "words" table - "lang" - varchar(2) - en-english, de-german, fr-french, etc. - "translatedWord" - text You can then assigned an endless list of examples, attachments and synonyms for “word”. In addition, you do not duplicate data in the case of synonyms ---------------------------------------------------------------------- bobby100 · 2024-09-02 14:47 Thank you all for the advices. The structure in paweld's example shows me another way of thinking (a good one). I'll give it a shot as the time permits, and I'll report back. ---------------------------------------------------------------------- alpine · 2024-09-02 15:26 Quote from: paweld on September 02, 2024, 07:52:09 amI would create more or less such a database structure:Code: [Select]table "types": - "typeId" - int autoincrement - "name" - varchar(50): verb, adjective etc. table "words": - "wordId" - int autoincrement - "word" - text - "???" - text - descriptive writing of the word capable of showing accents, ligatures (underscore the letters), silent letters (striked through) etc. - "typeId" - int - from "types" table - "description" - text table "samples": - "sampleId" - int autoincrement - "wordId" - int - from "words" table - "sample" - text table "attachments": - "attachmentId" - int autoincrement - "wordId" - int - from "words" table - "type" - char(1) - A-audio, G-graphic, V-video - "filename" - text - with path table "synonims": - "wordId" - int - from "words" table - "synonimWordId" - int - from "words" table table "translations": - "translateId" - int autoincrement - "wordId" - int - from "words" table - "lang" - varchar(2) - en-english, de-german, fr-french, etc. - "translatedWord" - text You can then assigned an endless list of examples, attachments and synonyms for “word”. In addition, you do not duplicate data in the case of synonyms It is much more complicated than that. See the attachment for a start: "types"-->"words" relationship cardinality isn't 1:N, it is M:N Good subject area knowledge is required, then comes the db modeling. My humble opinion. ---------------------------------------------------------------------- MarkMLl · 2024-09-02 15:35 Quote from: alpine on September 02, 2024, 05:26:23 pmGood subject area knowledge is required, then comes the db modeling. My humble opinion. I agree, plus not finding oneself in a position that one is fighting against the database. The backend facilities required to plan and then assemble the dictionary are likely to be substantially more than those required to actually run it. MarkMLl ---------------------------------------------------------------------- bobby100 · 2024-09-02 20:07 There is a standardized procedure for words with more than one meaning, or being of more than one type - each variant is a distinct word in a digital dictionary: - the words are stored like right1, right2, etc. - the numbers are not displayed to the reader The distinct variations have different sense (e.g. right from the example above), and there is also a category of subsense if the meaning of the word is subtly changed because of the content of the sentence, e.g. head and head of the family. I've spent last night reading about construction of digital dictionaries from some of the pioneers in this field: https://s3.amazonaws.com/downloads.sil.org/legacy/shoebox/MDF_2000.pdf There is FieldWorx Explorer, a modern version of the Shoebox: https://software.sil.org/fieldworks/ Going through the lectures from the FieldWorx website, one can learn a lot. As for the app, I miss some options, e.g. - I would like to have all the grammar declinations, conjugations etc on the screen with a chosen word. Perhaps it can be done in the program, but it isn't exported for the web version ---------------------------------------------------------------------- CM630 · 2024-09-02 20:43 If you want to learn databases that you should go with them. But if you want to do something about preserving (the memory of) your language - its another thing. Imho, in the second case you should do text format. Something like: wordtranscriptionmeaning onemeaning2 Or you could use some tags. Like wordtranscriptionmeaning oneverbmeaning2 I am not sure if tags would be usefull, paper dictionaries do not use them, computers dictionaries do not neither. About transcription there is IPA, no need to reinvent the wheel. The benefit ot using text format is that your documents might be used without you and after you. The future of Pascal in not bright. I have some hard experience with a family tree documented in an TurboPascal app by a cousin. ---------------------------------------------------------------------- Zvoni · 2024-09-03 08:57 hmm..... MAYBE as a first step: Try to reach out to something like "leo.org" Maybe they will give you some start-aid (how they do it), or maybe not. Or maybe they will agree to HOST your language..... ---------------------------------------------------------------------- Thaddy · 2024-09-03 12:58 I don't think this is about programming, but about a notational phonetic language. Such thing exists, is standardized and it is called IPA. E.g. wikipedia uses it always. https://en.wikipedia.org/wiki/Wikipedia:Manual_of_Style/Pronunciation. I think, for now, this is more helpful. IPA is a notational standard for pronunciation, although very confusing to me, but I am not a linguist. The way IPA notation works is a way to preserve a natural language or accent is spoken. So if you want to preserve a language that is only spoken by few, use IPA to document it and start conversations with linguists on how to do that. linguist scientists love that, so look up some contacts at universities in your area. You do not have a degree of any sort to trigger their interest. https://www.internationalphoneticalphabet.org/ Once you get up to speed with IPA, it is not unlikely we can help you to map most or all of it to Pascal to do text to speech and speech to text. We would need audio to detect the phonemes, though and map that to the notation. Phonemes and pronounciation differ, so we need both for expression. (happy, sad, laughing, disapproving and the like, basically how to express emotion) With the audio part I can help you.(slicing on many samples) ---------------------------------------------------------------------- MarkMLl · 2024-09-03 14:49 Quote from: Thaddy on September 03, 2024, 02:58:35 pmIPA is a notational standard for pronunciation, although very confusing to me, but I am not a linguist. At least two messages already mention it. However I think it's worth adding https://fumbling.it/posts/building-ipa-keyboard-part-four/ which discusses a physical IPA keyboard, which I suspect would end up much easier than the contortions needed to type it on a standard QWERTY. MarkMLl ---------------------------------------------------------------------- bobby100 · 2024-09-03 17:54 I am not ignoring IPA, but... until some software automatically converts audio to IPA, there is not much help for me there. Learning IPA from texts brings nothing. You need to hear something in order to make a link in your head between a sound and a symbol. Audio recording should be enough for now. As for a search for linguists - it is too much for a hobby project. About existing projects: https://www.paundurlic.com/vlaski.recnik/index.php - the problem here is that Mr. Durlic does not try to standardize the written form, but he rather uses just APHI (something like IPA, but not so widely used and also less detailed) - http://www.paundurlic.com/vlaski.recnik/sound.php . This means that every dialect gets its own written form of the same word http://www.gergina.org.rs/vlasko-pismo/ - this is a monstrosity - and it is recognized by the Serbian state as official Walachian alphabet (Slav people call us Walachian/Vlach/Vlah - https://en.wikipedia.org/wiki/Vlachs ). They added Walachian-specific letters to Serbian alphabet, both Latin and Cyrillic versions. It is something in the manner of how Russian state forced its own alphabet to Turkic nations in USSR - there is a bunch of letters that we do not need at all. Even the name of the website is written wrong - it should be Gherghina (flower dahlia), and not Gergina (h is a separator, not a sound). This is an example of incompetent people doing a bad job. The head of the organization and the inventor of this alphabet is a gynecologist. If I leave this job to him, you know where the future of this language is going... They aren't standardizing anything. They just write "by the feeling" for the purpose of organizing cultural manifestations (concerts, handcraft workshops and similar). ---------------------------------------------------------------------- CM630 · 2024-09-04 20:45 Your assumption that the Latin alphabeth is better than the Cyrillic alphabeth or the Armenian aphabeth for writing Romanian might be true, but might be wrong. Turkish and Albanian implemetation of Latin alphabeth are much, much better than the English one and even better than the German one. These implementation were done later, they were based on experience during the ages and have evaded some mistakes. Also, some decisions are made not because of linguistic reasons, but because of political ones. Just an example: "Craiova" is read "Krajova" or "Kraiova", but "ce face" is read "t͡ʃe fat͡ʃe". BTW, Romanian was written in Cyrillic until the middle of the ninghteenth century. Maybe caling the language from that time Romanianis wrong, I do not know. ---------------------------------------------------------------------- bobby100 · 2024-09-04 22:38 Quote from: CM630 on September 04, 2024, 10:45:25 pmAlso, some decisions are made not because of linguistic reasons, but because of political ones. Just an example: "Craiova" is read "Krajova" or "Kraiova", but "ce face" is read "t͡ʃe fat͡ʃe". C followed by E or by I isn't read C (K) anymore, but rather like English CH in Charlie. Same goes for G followed by E or by I - it is like G in Georgia, not like G in Bulgaria. It is in the nature of the language. I see that you know Cyrillic script, so I'll use some Cyrillic transcription. E.g. singular (un) Drac (a devil), transcribed to Cyrillic - Драк. The plural is built by adding an "I" at the end of the word - Draci (devils). Transcribed to Cyrillic - Драчи. Back to your example with "ce face" (what is he doing): Ieu fac, noi facem (I do, we do) - transcribed to Cyrillic - Иеу фак, нои фачем. Same for G - (un) Drag (a loved one) is with G like in Bulgaria, but the plural Dragi (loved ones) goes to G like in Georgia. So, in Craiova, C is followed by R, and you can read it like K, but in "ce face" you have two CE groups, and here happens the transformation. In standard Romanian, the letter H is used to break the transformation, like in word Cheie (a key). H isn't a vowel here, but is just used in script to depart E from C, so that the transformation does not happen. As for Romanian and Cyrillic script - yes, the church used it because of the influence of Russian/Ukrainian church, from where the religion came to Romania. The liturgy in churches were in Church-Slavic til 18th century. There wasn't a lot of literacy outside the churches. But, do not imagine Romanian Cyrillic script to be anything like modern Bulgarian, Russian or Serbian script, as it was derived from Church-Slavic Cyrillic script ( https://ro.wikipedia.org/wiki/Alfabetu l_limbii_rom%C3%A2ne#/media/Fi%C8%99ier:Romanian_Cyrillic_- _Lord's_Prayer_text.svg ). It contains letters like șt, în, ia, ie or ou - one letter for group of vowels. Latin alphabet for Romanian language is established in 1860. In Bucharest, in National Museum, you can also find tombstones written in Romanian, by using Greek alphabet. ---------------------------------------------------------------------- CM630 · 2024-09-06 18:36 Quote from: bobby100 on September 05, 2024, 12:38:06 am... C followed by E or by I isn't read C (K) anymore, but rather like English CH in Charlie. Same goes for G followed by E or by I - it is like G in Georgia, not like G in Bulgaria. It is in the nature of the language. ... I am aware of that, that is why I gave it as an example. But the nature of the languge is one thing, while the alphabeth (which is much more artificial) is another thing. Your own examples show the ambiguous usage of „c“ and „g“, which might not be present in the yugoslavian script (I believe you know whether it is so or not, I am just assuming). Romanian history does not seem to be in your area of interest, but maybe you would find interesting that the Serbian letter „Џ“ is actially an adopted Romanian/ Wallach letter for the same sound. Since we have got far away from Lazarus, I think I shall say nothing more in thus thread. ---------------------------------------------------------------------- MarkMLl · 2024-09-06 19:39 Quote from: CM630 on September 06, 2024, 08:36:16 pmI am aware of that, that is why I gave it as an example. But the nature of the languge is one thing, while the alphabeth (which is much more artificial) is another thing. All of which reinforce the concept of Notation as a Tool of Thought: I believe that one can only get so far with stored audio snippets. MarkMLl