Commit Graph

243 Commits

Author SHA1 Message Date
toasted-nutbread
a81787c55b
Remove the dictionary tag from the definitionTags (#1324)
* Remove dictionary tag from term definitionTags and kanji tags

* Update test data

* Update templates

* Update tests

* Update information
2021-01-28 21:33:30 -05:00
toasted-nutbread
b20622b2c8
Core refactor (#1207)
* Copy set intersection functions

* Remove unused functions

* Simplify url check

* Remove parseUrl

* Simplify stringReverse

* Remove hasOwn due to infrequent use

* Rename errorToJson/jsonToError to de/serializeError

For clarity on intended use.

* Fix time argument on timeout

* Add missing return value

* Throw an error for unexpected argument values

* Add documentation comments
2021-01-07 21:36:20 -05:00
toasted-nutbread
8e304b83c6
Translator regex replacements (#1199)
* Add support for regex replacements during the translation process

* Allow assignment of textReplacements

* Rename

* Set up test data

* Write expected data

* Set up options

* Prevent infinite loop if regex matches empty string

* Implement setting controller

* Add support for testing pattern replacements
2021-01-03 12:12:55 -05:00
toasted-nutbread
41f6de3e3a
Simplify collapseEmphaticOptions options (#1196) 2021-01-01 23:16:44 -05:00
toasted-nutbread
a02891e52c
Update copyright year (#1194)
* Update copyright year

* Remove redundant Author info
2021-01-01 14:50:41 -05:00
toasted-nutbread
4650f0b542
Translator frequency data improvements (#1184)
* Specify if a frequency has a specific reading

* Make frequency data creation more standardized

* Update test data
2020-12-30 15:12:08 -05:00
toasted-nutbread
af9b4d2b93
Update termTags for termGrouped mode to include all definition tags (#1138)
* Update termTags for termGrouped mode to include all definition tags

* Update tests
2020-12-19 16:47:32 -05:00
toasted-nutbread
2910685a96
Database test updates (#1086)
* Remove unused

* Create DatabaseVM

* Update sequences in test dictionary

* Update documentation

* Add second valid testing dictionary

* Set up translator tests

* Write expected data
2020-12-08 20:27:36 -05:00
toasted-nutbread
34451ebf71
JapaneseUtil normalization (#1076)
* Use JapaneseUtil as a class which is manually instantiated

* Use alias function for toKana
2020-11-29 13:09:02 -05:00
toasted-nutbread
496af0b5f2
Fix incorrect frequency field being used (#1059) 2020-11-24 11:55:25 -05:00
toasted-nutbread
112f08a13e
Fix term details not using the expanded tags (#1028) 2020-11-14 12:09:14 -05:00
toasted-nutbread
ec021964b7
Compact tags refactor (#1021)
* Update translator to flag redundant tags instead of remove

* Update how compact tags are shown in the popup

* Pass compactTags option to note builder

* Update options templates

* Add options upgrade

* Add options upgrade test
2020-11-12 20:34:11 -05:00
toasted-nutbread
a8ff38eec3
Fix display issues (#984)
* Remove unused function

* Update expresionMulti detection

* Simplify kanji early escape

* Simplify frequency/pitch data creation

* Update implementation of _buildTermMeta

* Update how pitch accents sources are collected

* Remove old _buildTermMeta
2020-11-01 21:24:35 -05:00
toasted-nutbread
cee64f91eb
Fix data inconsistencies (#891)
* Simplify variable name

* Add dictionaryNames array

* Update how the dictionary field is assigned

* Use "dictionary" instead of "title" for consistency

* Remove redundant dictionary name

* Use _expandTags to create pitch accent tags
2020-10-06 20:28:49 -04:00
toasted-nutbread
69f94f8cb8
Update merged definitions to have consistent structure (#890)
* Update comment about definitions type

* Add conversion to ensure consistent data structure
2020-10-06 19:31:53 -04:00
toasted-nutbread
7ce87c51b4
Don't add duplicate definitions (#889) 2020-10-05 22:50:10 -04:00
toasted-nutbread
0cfaa1a5a8
Dictionary definition structure fixes (#888)
* Restore support for tags without metadata

* Fix missing spread
2020-10-05 21:10:52 -04:00
toasted-nutbread
8c8c27001e
Fix incorrect definitions array being used for compatibility merged definitions (#887) 2020-10-04 22:21:16 -04:00
toasted-nutbread
54810510fa
Fix primary audio source (#886)
* Add abstraction: _getDefinitionPrimaryExpressionAndReading

* Reuse existing definitions in a sequence

* Revert change to related definition source/rawSource/sourceTerm

* Update _getDefinitionPrimaryExpressionAndReading to return best match
2020-10-04 22:04:44 -04:00
toasted-nutbread
2d3cf89b49
Fix doc comments for translation options (#885) 2020-10-04 19:36:21 -04:00
toasted-nutbread
c604a406f0
Translator organization (#884)
* Group reduction functions together

* Group simple helpers

* Add label comments

* Move core _findTerms functions to top

* Move _getSecondarySearchDictionaryMap and _getDictionaryPriority

* Move tag helpers

* Move _getTextOptionEntryVariants

* Move functions up

* Move functions up

* Update label comments
2020-10-04 19:33:22 -04:00
toasted-nutbread
8b8806484e
Add check to verify expressions are strings (#883) 2020-10-04 19:04:21 -04:00
toasted-nutbread
f904b3e11a
Add sourceTermExactMatchCount to prioritize exact expression matches (#882) 2020-10-04 18:54:03 -04:00
toasted-nutbread
561e36e88d
Translator definition source term (and other info) (#881)
* Add sourceTerm field to 'term' and 'termGrouped' definitions

* Fix comparison of expressions which are always the same

* Rename/restructure term info map

* Add source term

* Add sourceTerm to expressions array

* Reuse furiganaSegments

* Add helper function _createExpressionDetailsListFromTermInfoMap

* Add expressions array to termMergedByGlossary

* Add expressions to definition types 'term' and 'termGrouped'

* Rename expressionDetails* to termDetails*

* Correct the source/rawSource/sourceTerm for related sequenced definitions

* Simplify structure of sequencedDefinitions

* Remove TODO
2020-10-04 18:10:10 -04:00
toasted-nutbread
7e31dcca39
Move deinflection reasons setup to backend (#880)
* Move deinflection reasons setup to backend

* Remove async
2020-10-04 13:09:04 -04:00
toasted-nutbread
2bd82353e4
Translator options refactor (#879)
* Refactor internal options for findTerms to not use the settings object

* Move findTerms/findKanji options creation

* Deconstruct used options values to variables before any await calls

* Rename findTermsOptions to just options

* Add documentation comments

* Add type information about definitions
2020-10-04 12:54:55 -04:00
toasted-nutbread
86c64ac4c2
Translator refactoring (#878)
* Add helper function _getSecondarySearchDictionaryMap

* Use spreads instead of concat

* Create helper function for assigning disambiguation info

* Add _getTermTagsScoreSum helper

* Rename variables

* Move assignment of definition details

* Convert databaseDefinitions to standard definitions

* Move creation of definitions

* Always pass definitionsByGlossary directly to _mergeByGlossary

* Refactor how used definitions are tracked

* Change _mergeByGlossary parameters

* Simplify _mergeByGlossary

* Generate definition structure at the end of the function

* Add reusable function to clone a tag

* Restructure merged sub-definition creation

* Move setup of totalExpressionSet/totalReadingSet

* Track all definitions with matching glossary

* Move creation of definitionTags

* Update score calculation for multiple definitions

* Disambiguate function name

* Rename variables

* Add helper function _createMergedGlossaryTermDefinition

* Remove rarely used check for secondary search results

Unclear what the purpose was since it was almost never used

* Update data structure

* Rename function/variable

* Add helper functions _createMergedTermDefinition

* Assign expressionDetails to variable

* Add rawSource to grouped definitions

* Use shared function for creating unused definition entries

* Add helper function _createMapKey

* Refactor _createExpressionDetails

* Change argument order

* Add _getTagNamesWithCategory helper

* Optimize _compressDefinitionTags

* Add _createGroupedTermDefinition helper

* Add type information

* Use consistent order for term definition fields, add comments for omitted fields

* Remove unused index field

* Add helper function _createKanjiDefinition

* Update types

* Add frequencies/pitches arrays

* Update terms array setup

* Simplify for loop

* Remove redundant property assignment

* Simplify how deinflections with no definitions are ignored

* Ensure duplicates are removed

* Rename variable

* Add comments about duplicate data

* Refactor variable/parameter names

* Move functions

* Rename functions

* Rename variable

* Add dictionaryPriority field

* Simplify priority comparison

* Rename argument

* Optimize comparison function when dictionary priority is not needed

* Add length check

* Remove unused parameter

* Rename variables/parameters

* Rename dictionaries to enabledDictionaryMap

* Store compactTags/mainDictionary options to variables

* Refactor _getSearchableText
2020-10-04 11:12:15 -04:00
toasted-nutbread
ef333b6d72
Dictionary data structure improvements (#877)
* Simplify object structure of deinflections

* Modify existing array when removing duplicates

* Update _sortDefinitions to not return a value

* Use explicit argument value

* Use a Collator for string comparisons

* Simplify order comparison

* Improve tag creation and sanitization

* Switch .map to a for loop, skip null meta

* Add _createKanjiStat

* Move _sortDefinitions definition

* Fix kanji stat sorting

* Remove return value from _sortTags

* Add _sortKanji

* Add fast exits for sorting

* Add _sortDefinitionsByIndex

* Combine sort function

* Improve creation of kanji definition data

* Use "databaseDefinitions" instead of "definitions" to disambiguate

* Simplify

* Simplify further

* Simplify max length calculation

* More destructuring

* Use databaseDefinitions variable name

* Move _mergeBySequence body into _getSequencedDefinitions

* Use databaseDefinitions field name

* Move maxLength calculation

* Use shared _createTermDefinitionFromDatabaseDefinition

* Simplify map

* Move definition sorts for better consistency and less redundancy

* Move _createExpression function

* Update setup of sequenced definitions

* Simplify for loop
2020-10-02 17:59:14 -04:00
toasted-nutbread
9f8f83508e
Merge dictionary.js functions into Translator class (#726)
* Move dictFieldSplit

* Copy dictionary.js definitions into translator.js

* Convert to member functions

* Refactor names

* Remove dictionary.js

* Rename
2020-08-09 21:04:09 -04:00
toasted-nutbread
480e0e15e3
Translator refactor (#720)
* Move simple find sort into translator.js

* Remove unused

* Use direct reference to database

* Make field private

* Remove statics

* Convert functions to private

* Organize by public/private
2020-08-09 13:21:14 -04:00
toasted-nutbread
b1b33f8beb
Fix fetch requests (#708)
* Revert audio fetching functionality to use XMLHttpRequest

* Replace requestJson

* Replace requestJson

* Replace requestJson

* Replace requestJson and requestText

* Fix tests

* Include support for vulgar word searches

* Remove request.js
2020-08-02 13:30:55 -04:00
toasted-nutbread
021ccb5ac3
Move util database modification functions (#499)
* Update onProgress callback to handle multiple arguments

* Add apiImportDictionaryArchive

* Add apiDeleteDictionary

* Make onProgress the last argument for consistency

* Remove deprecated util functions

* Fix issue with missing progress args

* Remove function calls which modify the database from Translator

* Update tests

* Fix errors not being serialized correctly in _createActionListenerPort
2020-05-06 19:28:26 -04:00
toasted-nutbread
9297eb45ae
Merge pull request #450 from toasted-nutbread/frequency-readings
Add support for filtering frequency metadata based on readings
2020-04-18 14:14:30 -04:00
toasted-nutbread
649adb13d8
Merge pull request #440 from toasted-nutbread/collapse-emphatic-sequences
Add support for collapsing emphatic character sequences
2020-04-12 11:42:46 -04:00
toasted-nutbread
ceb12ac415 Add support for filtering frequency metadata based on readings 2020-04-11 23:20:36 -04:00
toasted-nutbread
70f0b8b0cd Fix 'full' mode not being a superset of 'true' mode 2020-04-11 18:58:14 -04:00
4e5029f7ec Update license author field to broader to explicitly reflect that
Yomichan is an ongoing community project. It really does take a village,
and everyone who contributes code owns a part of it :)
2020-04-10 11:10:21 -07:00
toasted-nutbread
42a2917bf7 Add support for collapsing emphatic character sequences 2020-04-10 11:56:18 -04:00
toasted-nutbread
3684a479c5
Merge pull request #424 from toasted-nutbread/dictionary-importer
Dictionary importer
2020-04-05 12:52:07 -04:00
toasted-nutbread
058f626efd
Merge pull request #423 from toasted-nutbread/text-source-map
Text source map
2020-04-05 12:51:56 -04:00
toasted-nutbread
c193a703cc Move database creation into Backend 2020-03-30 20:39:04 -04:00
toasted-nutbread
ae84d13757 Create simplified source map class 2020-03-28 18:17:00 -04:00
toasted-nutbread
047efaa3db Add support for returning pitch data from the database 2020-03-28 10:25:57 -04:00
toasted-nutbread
248a18dd72 Fix case issue 2020-03-15 22:24:30 -04:00
toasted-nutbread
0f15cca2df Convert Japanese utilities to a module-like style 2020-03-15 22:24:30 -04:00
toasted-nutbread
64fc0349a1 Update global declarations 2020-03-10 22:40:35 -04:00
toasted-nutbread
72219ba353 Replace charCodeAt and fromCharCode with codePointAt and fromCodePoint 2020-03-09 21:58:06 -04:00
siikamiika
9af8f999f4 compatibility transform non-primary definitions 2020-03-01 21:26:45 +02:00
toasted-nutbread
54bb702b84 Updates for PR feedback 2020-02-22 14:43:06 -05:00
toasted-nutbread
69b5007842 Reduce number of redundant Map.get calls in dictTermsMergeByGloss 2020-02-22 14:43:06 -05:00