diff --git a/README.md b/README.md index 6176e12f..e29069b7 100644 --- a/README.md +++ b/README.md @@ -258,52 +258,72 @@ versions packaged. ## Frequently Asked Questions ## -* **Why does the Kanji results page display "No data found" for several fields?** +**I'm having problems importing dictionaries in Firefox, what do I do?** - You are using data from the KANJIDIC dictionary that was exported for an earlier version of Yomichan. It does not - contain the additional information which newer versions of Yofomichan expect. Unfortunately, since major browser - implementations of IndexedDB do not provide reliable means for selective bulk data deletion, you will need purge - your database and install the latest version of the KANJIDIC to see additional information about characters. +Yomichan uses the cross-browser IndexedDB system for storing imported dictionary data into your user profile. Although +everything "just works" in Chrome, depending on settings, Firefox users can run into problems due browser bugs. +Yomichan catches errors and tries to offer suggestions about how to work around Firefox issues, but in general at least +one of the following solutions should work for you: -* **Can I still create cards without HTML formatting? The option for it is gone!** +* Make sure you have cookies enabled. It appears that disabling them also disables IndexedDB for some reason. You + can still have cookies be disabled on other sites; just make sure to add the Yomichan extension to the whitelist of + whatever tool you are using to restrict cookies. You can get the extension "URL" by looking at the address bar when + you have the search page open. +* Make sure that you have sufficient disk space available on the drive Firefox uses to store your user profile. + Firefox limits the amount of space that can be used by IndexedDB to a small fraction of the disk space actually + available on your computer. +* Make sure that you have history set to "Remember history" enabled in your privacy settings. When this option is + set to "Never remember history", IndexedDB access is once again disabled for an inexplicable reason. +* As a last resort, try using the [Refresh Firefox](https://support.mozilla.org/en-US/kb/reset-preferences-fix-problems) + feature to reset your user profile. It appears that the Firefox profile system can corrupt itself preventing + IndexedDB from being accessible to Yomichan. - Developing Yomichan is a constant balance between including useful features and keeping complexity at a minimum. - With the new user-editable card template system, it is possible to create text-only cards without having to double - the number field of templates in the extension itself. If you would like to stop HTML tags from being added to your - cards, simply copy the contents of the text-only field template into the template box on - the Anki settings page (make sure you have the *Show advanced options* checkbox ticked), making sure to replace the - existing values. +**Why does the Kanji results page display "No data found" for several fields?** -* **Will you add support for online dictionaries?** +You are using data from the KANJIDIC dictionary that was exported for an earlier version of Yomichan. It does not +contain the additional information which newer versions of Yofomichan expect. Unfortunately, since major browser +implementations of IndexedDB do not provide reliable means for selective bulk data deletion, you will need purge +your database and install the latest version of the KANJIDIC to see additional information about characters. - Online dictionaries will never be implemented because it is impossible to support them in a robust way. In order to - perform Japanese deinflection, Yomichan must execute dozens of database queries per every single word. Factoring in - network latency and the fragility of web scraping, I do not believe that it is possible to realize a good user - experience. +**Can I still create cards without HTML formatting? The option for it is gone!** -* **Is it possible to use Yomichan with files saved locally on my computer with Chrome?** +Developing Yomichan is a constant balance between including useful features and keeping complexity at a minimum. +With the new user-editable card template system, it is possible to create text-only cards without having to double +the number field of templates in the extension itself. If you would like to stop HTML tags from being added to your +cards, simply copy the contents of the text-only field template into the template box on +the Anki settings page (make sure you have the *Show advanced options* checkbox ticked), making sure to replace the +existing values. - In order to use Yomichan with local files in Chrome, you must first tick the *Allow access to file URLs* checkbox - for Yomichan on the extensions page. Due to the restrictions placed on browser addons in the WebExtensions model, it - will likely never be possible to use Yomichan with PDF files. +**Will you add support for online dictionaries?** -* **Is it possible to delete individual dictionaries without purging the database?** +Online dictionaries will never be implemented because it is impossible to support them in a robust way. In order to +perform Japanese deinflection, Yomichan must execute dozens of database queries per every single word. Factoring in +network latency and the fragility of web scraping, I do not believe that it is possible to realize a good user +experience. - Although it is technically possible to purge specific dictionaries, due to the limitations of the underlying browser - IndexedDB system, this process is *extremely* slow. For example, it can take up to ten minutes to delete a single - moderately-sized term dictionary! Instead of including a borderline unusable feature in Yomichan, I have chosen to - disable dictionary deletion entirely. +**Is it possible to use Yomichan with files saved locally on my computer with Chrome?** -* **Why aren't EPWING dictionaries bundled with Yomichan?** +In order to use Yomichan with local files in Chrome, you must first tick the *Allow access to file URLs* checkbox +for Yomichan on the extensions page. Due to the restrictions placed on browser addons in the WebExtensions model, it +will likely never be possible to use Yomichan with PDF files. - The vast majority of EPWING dictionaries are proprietary, so unfortunately I am unable to legally include them in - this extension for copyright reasons. +**Is it possible to delete individual dictionaries without purging the database?** -* **When are you going to add support for $MYLANGUAGE?** +Although it is technically possible to purge specific dictionaries, due to the limitations of the underlying browser +IndexedDB system, this process is *extremely* slow. For example, it can take up to ten minutes to delete a single +moderately-sized term dictionary! Instead of including a borderline unusable feature in Yomichan, I have chosen to +disable dictionary deletion entirely. - Developing Yomichan requires a significant understanding of Japanese sentence structure and grammar. I have no time - to invest in learning yet another language; therefore other languages will not be supported. I will also not accept - pull request containing this functionality, as I will ultimately be the one maintaining your code. +**Why aren't EPWING dictionaries bundled with Yomichan?** + +The vast majority of EPWING dictionaries are proprietary, so unfortunately I am unable to legally include them in +this extension for copyright reasons. + +**When are you going to add support for $MYLANGUAGE?** + +Developing Yomichan requires a significant understanding of Japanese sentence structure and grammar. I have no time +to invest in learning yet another language; therefore other languages will not be supported. I will also not accept +pull request containing this functionality, as I will ultimately be the one maintaining your code. ## Screenshots ## diff --git a/ext/bg/js/api.js b/ext/bg/js/api.js index 9f65bb07..de3ad64e 100644 --- a/ext/bg/js/api.js +++ b/ext/bg/js/api.js @@ -29,9 +29,11 @@ async function apiTermsFind(text) { const options = utilBackend().options; const translator = utilBackend().translator; - const searcher = options.general.groupResults ? - translator.findTermsGrouped.bind(translator) : - translator.findTermsSplit.bind(translator); + const searcher = { + 'merge': translator.findTermsMerged, + 'split': translator.findTermsSplit, + 'group': translator.findTermsGrouped + }[options.general.resultOutputMode].bind(translator); const {definitions, length} = await searcher( text, diff --git a/ext/bg/js/audio.js b/ext/bg/js/audio.js index ce47490c..549288f5 100644 --- a/ext/bg/js/audio.js +++ b/ext/bg/js/audio.js @@ -140,8 +140,13 @@ async function audioInject(definition, fields, mode) { } try { - const url = await audioBuildUrl(definition, mode); - const filename = audioBuildFilename(definition); + let audioSourceDefinition = definition; + if (definition.hasOwnProperty('expressions')) { + audioSourceDefinition = definition.expressions[0]; + } + + const url = await audioBuildUrl(audioSourceDefinition, mode); + const filename = audioBuildFilename(audioSourceDefinition); if (url && filename) { definition.audio = {url, filename}; diff --git a/ext/bg/js/database.js b/ext/bg/js/database.js index 6ceb3ec8..3c7f6aab 100644 --- a/ext/bg/js/database.js +++ b/ext/bg/js/database.js @@ -40,6 +40,9 @@ class Database { kanjiMeta: '++,dictionary,character', tagMeta: '++,dictionary,name' }); + this.db.version(4).stores({ + terms: '++id,dictionary,expression,reading,sequence' + }); await this.db.open(); } @@ -68,12 +71,66 @@ class Database { results.push({ expression: row.expression, reading: row.reading, - tags: dictFieldSplit(row.tags), + definitionTags: dictFieldSplit(row.definitionTags || row.tags || ''), + termTags: dictFieldSplit(row.termTags || ''), rules: dictFieldSplit(row.rules), glossary: row.glossary, score: row.score, dictionary: row.dictionary, - id: row.id + id: row.id, + sequence: typeof row.sequence === 'undefined' ? -1 : row.sequence + }); + } + }); + + return results; + } + + async findTermsExact(term, reading, titles) { + if (!this.db) { + throw 'Database not initialized'; + } + + const results = []; + await this.db.terms.where('expression').equals(term).each(row => { + if (row.reading === reading && titles.includes(row.dictionary)) { + results.push({ + expression: row.expression, + reading: row.reading, + definitionTags: dictFieldSplit(row.definitionTags || row.tags || ''), + termTags: dictFieldSplit(row.termTags || ''), + rules: dictFieldSplit(row.rules), + glossary: row.glossary, + score: row.score, + dictionary: row.dictionary, + id: row.id, + sequence: typeof row.sequence === 'undefined' ? -1 : row.sequence + }); + } + }); + + return results; + } + + async findTermsBySequence(sequence, mainDictionary) { + if (!this.db) { + throw 'Database not initialized'; + } + + const results = []; + await this.db.terms.where('sequence').equals(sequence).each(row => { + if (row.dictionary === mainDictionary) { + results.push({ + expression: row.expression, + reading: row.reading, + definitionTags: dictFieldSplit(row.definitionTags || row.tags || ''), + termTags: dictFieldSplit(row.termTags || ''), + rules: dictFieldSplit(row.rules), + glossary: row.glossary, + score: row.score, + dictionary: row.dictionary, + id: row.id, + sequence: typeof row.sequence === 'undefined' ? -1 : row.sequence }); } }); @@ -163,7 +220,7 @@ class Database { return result; } - async getTitles() { + async summarize() { if (this.db) { return this.db.dictionaries.toArray(); } else { @@ -177,7 +234,7 @@ class Database { } const indexDataLoaded = async summary => { - if (summary.version > 2) { + if (summary.version > 3) { throw 'Unsupported dictionary version'; } @@ -196,11 +253,11 @@ class Database { const rows = []; if (summary.version === 1) { - for (const [expression, reading, tags, rules, score, ...glossary] of entries) { + for (const [expression, reading, definitionTags, rules, score, ...glossary] of entries) { rows.push({ expression, reading, - tags, + definitionTags, rules, score, glossary, @@ -208,14 +265,16 @@ class Database { }); } } else { - for (const [expression, reading, tags, rules, score, glossary] of entries) { + for (const [expression, reading, definitionTags, rules, score, glossary, sequence, termTags] of entries) { rows.push({ expression, reading, - tags, + definitionTags, rules, score, glossary, + sequence, + termTags, dictionary: summary.title }); } @@ -300,12 +359,13 @@ class Database { } const rows = []; - for (const [name, category, order, notes] of entries) { + for (const [name, category, order, notes, score] of entries) { const row = dictTagSanitize({ name, category, order, notes, + score, dictionary: summary.title }); @@ -350,12 +410,11 @@ class Database { const summary = { title: index.title, revision: index.revision, + sequenced: index.sequenced, version: index.format || index.version }; - if (indexDataLoaded) { - await indexDataLoaded(summary); - } + await indexDataLoaded(summary); const buildTermBankName = index => `term_bank_${index + 1}.json`; const buildTermMetaBankName = index => `term_meta_bank_${index + 1}.json`; @@ -390,7 +449,7 @@ class Database { const bank = []; for (const name in index.tagMeta) { const tag = index.tagMeta[name]; - bank.push([name, tag.category, tag.order, tag.notes]); + bank.push([name, tag.category, tag.order, tag.notes, tag.score]); } tagDataLoaded(summary, bank, ++bankTotalCount, bankLoadedCount++); diff --git a/ext/bg/js/dictionary.js b/ext/bg/js/dictionary.js index 57acbe5e..fea5f3e5 100644 --- a/ext/bg/js/dictionary.js +++ b/ext/bg/js/dictionary.js @@ -89,7 +89,7 @@ function dictTermsSort(definitions, dictionaries=null) { return 1; } - return v2.expression.localeCompare(v1.expression); + return v2.expression.toString().localeCompare(v1.expression.toString()); }); } @@ -110,6 +110,33 @@ function dictTermsUndupe(definitions) { return definitionsUnique; } +function dictTermsCompressTags(definitions) { + let lastDictionary = ''; + let lastPartOfSpeech = ''; + + for (const definition of definitions) { + const dictionary = JSON.stringify(definition.definitionTags.filter(tag => tag.category === 'dictionary').map(tag => tag.name).sort()); + const partOfSpeech = JSON.stringify(definition.definitionTags.filter(tag => tag.category === 'partOfSpeech').map(tag => tag.name).sort()); + + const filterOutCategories = []; + + if (lastDictionary === dictionary) { + filterOutCategories.push('dictionary'); + } else { + lastDictionary = dictionary; + lastPartOfSpeech = ''; + } + + if (lastPartOfSpeech === partOfSpeech) { + filterOutCategories.push('partOfSpeech'); + } else { + lastPartOfSpeech = partOfSpeech; + } + + definition.definitionTags = definition.definitionTags.filter(tag => !filterOutCategories.includes(tag.category)); + } +} + function dictTermsGroup(definitions, dictionaries) { const groups = {}; for (const definition of definitions) { @@ -136,6 +163,7 @@ function dictTermsGroup(definitions, dictionaries) { expression: firstDef.expression, reading: firstDef.reading, reasons: firstDef.reasons, + termTags: groupDefs[0].termTags, score: groupDefs.reduce((p, v) => v.score > p ? v.score : p, Number.MIN_SAFE_INTEGER), source: firstDef.source }); @@ -144,6 +172,116 @@ function dictTermsGroup(definitions, dictionaries) { return dictTermsSort(results); } +function dictTermsMergeBySequence(definitions, mainDictionary) { + const definitionsBySequence = {'-1': []}; + for (const definition of definitions) { + if (mainDictionary === definition.dictionary && definition.sequence >= 0) { + if (!definitionsBySequence[definition.sequence]) { + definitionsBySequence[definition.sequence] = { + reasons: definition.reasons, + score: Number.MIN_SAFE_INTEGER, + expression: new Set(), + reading: new Set(), + expressions: new Map(), + source: definition.source, + dictionary: definition.dictionary, + definitions: [] + }; + } + const score = Math.max(definitionsBySequence[definition.sequence].score, definition.score); + definitionsBySequence[definition.sequence].score = score; + } else { + definitionsBySequence['-1'].push(definition); + } + } + + return definitionsBySequence; +} + +function dictTermsMergeByGloss(result, definitions, appendTo, mergedIndices) { + const definitionsByGloss = appendTo || {}; + for (const [index, definition] of definitions.entries()) { + if (appendTo) { + let match = false; + for (const expression of result.expressions.keys()) { + if (definition.expression === expression) { + for (const reading of result.expressions.get(expression).keys()) { + if (definition.reading === reading) { + match = true; + break; + } + } + } + if (match) { + break; + } + } + + if (!match) { + continue; + } else if (mergedIndices) { + mergedIndices.add(index); + } + } + + const gloss = JSON.stringify(definition.glossary.concat(definition.dictionary)); + if (!definitionsByGloss[gloss]) { + definitionsByGloss[gloss] = { + expression: new Set(), + reading: new Set(), + definitionTags: [], + glossary: definition.glossary, + source: result.source, + reasons: [], + score: definition.score, + id: definition.id, + dictionary: definition.dictionary + }; + } + + definitionsByGloss[gloss].expression.add(definition.expression); + definitionsByGloss[gloss].reading.add(definition.reading); + + result.expression.add(definition.expression); + result.reading.add(definition.reading); + + // result->expressions[ Expression1[ Reading1[ Tag1, Tag2 ] ], Expression2, ... ] + if (!result.expressions.has(definition.expression)) { + result.expressions.set(definition.expression, new Map()); + } + if (!result.expressions.get(definition.expression).has(definition.reading)) { + result.expressions.get(definition.expression).set(definition.reading, new Set()); + } + + for (const tag of definition.definitionTags) { + if (!definitionsByGloss[gloss].definitionTags.find(existingTag => existingTag.name === tag.name)) { + definitionsByGloss[gloss].definitionTags.push(tag); + } + } + + for (const tag of definition.termTags) { + result.expressions.get(definition.expression).get(definition.reading).add(tag); + } + } + + for (const gloss in definitionsByGloss) { + const definition = definitionsByGloss[gloss]; + definition.only = []; + if (!utilSetEqual(definition.expression, result.expression)) { + for (const expression of utilSetIntersection(definition.expression, result.expression)) { + definition.only.push(expression); + } + } + if (!utilSetEqual(definition.reading, result.reading)) { + for (const reading of utilSetIntersection(definition.reading, result.reading)) { + definition.only.push(reading); + } + } + } + + return definitionsByGloss; +} + function dictTagBuildSource(name) { return dictTagSanitize({name, category: 'dictionary', order: 100}); } @@ -153,6 +291,7 @@ function dictTagSanitize(tag) { tag.category = tag.category || 'default'; tag.notes = tag.notes || ''; tag.order = tag.order || 0; + tag.score = tag.score || 0; return tag; } @@ -207,10 +346,12 @@ async function dictFieldFormat(field, definition, mode, options) { const data = { marker, definition, - group: options.general.groupResults, + group: options.general.resultOutputMode === 'group', + merge: options.general.resultOutputMode === 'merge', modeTermKanji: mode === 'term-kanji', modeTermKana: mode === 'term-kana', - modeKanji: mode === 'kanji' + modeKanji: mode === 'kanji', + compactGlossaries: options.general.compactGlossaries }; const html = await apiTemplateRender(options.anki.fieldTemplates, data, true); diff --git a/ext/bg/js/options.js b/ext/bg/js/options.js index 36ab7694..9f1414ad 100644 --- a/ext/bg/js/options.js +++ b/ext/bg/js/options.js @@ -19,12 +19,26 @@ function optionsFieldTemplates() { return ` + {{#*inline "glossary-single"}} {{~#unless brief~}} - {{~#if tags~}}({{#each tags}}{{name}}{{#unless @last}}, {{/unless}}{{/each}}) {{/if~}} + {{~#if definitionTags~}}({{#each definitionTags}}{{name}}{{#unless @last}}, {{/unless}}{{/each}}) {{/if~}} + {{~#if only~}}({{#each only}}{{{.}}}{{#unless @last}}, {{/unless}}{{/each}} only) {{/if~}} {{~/unless~}} {{~#if glossary.[1]~}} -
This dictionary is outdated and may not support new extension features; please import the latest version.
\n"; +},"3":function(container,depth0,helpers,partials,data) { return "checked"; },"compiler":[7,">= 4.0.0"],"main":function(container,depth0,helpers,partials,data) { var stack1, helper, alias1=depth0 != null ? depth0 : (container.nullContext || {}), alias2=helpers.helperMissing, alias3="function", alias4=container.escapeExpression; @@ -11,9 +13,13 @@ templates['dictionary.html'] = template({"1":function(container,depth0,helpers,p + alias4(((helper = (helper = helpers.title || (depth0 != null ? depth0.title : depth0)) != null ? helper : alias2),(typeof helper === alias3 ? helper.call(alias1,{"name":"title","hash":{},"data":data}) : helper))) + " rev." + alias4(((helper = (helper = helpers.revision || (depth0 != null ? depth0.revision : depth0)) != null ? helper : alias2),(typeof helper === alias3 ? helper.call(alias1,{"name":"revision","hash":{},"data":data}) : helper))) - + "\n\n