![\"\u041c\u0430\u043b\u044e\u043d\u0430\u043a](\"https:\/\/ssrlab.by\/wp-content\/uploads\/2017\/08\/Tokenizer_screen01.png\")
<\/a>Tokenizer\"><\/a>
\u041c\u0430\u043b\u044e\u043d\u0430\u043a 1. \u0413\u0440\u0430\u0444\u0456\u0447\u043d\u044b \u0456\u043d\u0442\u044d\u0440\u0444\u0435\u0439\u0441 \u0441\u044d\u0440\u0432\u0456\u0441\u0443 \u00ab\u0422\u0430\u043a\u0435\u043d\u0456\u0437\u0430\u0442\u0430\u0440\u00bb<\/p><\/div>\n
\u041a\u0430\u0440\u044b\u0441\u0442\u0430\u043b\u044c\u043d\u0456\u0446\u043a\u0456 \u0441\u0446\u044d\u043d\u0430\u0440 \u043f\u0440\u0430\u0446\u044b \u0437 \u0441\u044d\u0440\u0432\u0456\u0441\u0430\u043c<\/strong><\/p>\n \u041c\u0430\u043b\u044e\u043d\u0430\u043a 2. \u0412\u044b\u043d\u0456\u043a\u0456 \u043f\u0440\u0430\u0446\u044b \u0441\u044d\u0440\u0432\u0456\u0441\u0430 \u00ab\u0422\u0430\u043a\u0435\u043d\u0456\u0437\u0430\u0442\u0430\u0440\u00bb: \u0432\u044b\u043b\u0443\u0447\u0430\u043d\u044b\u044f \u0442\u043e\u043a\u0435\u043d\u044b<\/p><\/div>\n <\/p>\n \u0414\u043e\u0441\u0442\u0443\u043f \u0434\u0430 \u0441\u044d\u0440\u0432\u0456\u0441\u0430 \u043f\u0440\u0430\u0437 API<\/strong><\/p>\n \u0414\u043b\u044f \u0434\u043e\u0441\u0442\u0443\u043f\u0443 \u0434\u0430 \u0441\u044d\u0440\u0432\u0456\u0441\u0430 \u201c\u0422\u0430\u043a\u0435\u043d\u0456\u0437\u0430\u0442\u0430\u0440\u201d \u043f\u0440\u0430\u0437 API, \u043d\u0435\u0430\u0431\u0445\u043e\u0434\u043d\u0430 \u0430\u0434\u043f\u0440\u0430\u0432\u0456\u0446\u044c AJAX-\u0437\u0430\u043f\u044b\u0442 \u0442\u044b\u043f\u0443 POST \u043d\u0430 \u0430\u0434\u0440\u0430\u0441 https:\/\/corpus.by\/Tokenizer\/api.php. \u041f\u0440\u0430\u0437 \u043c\u0430\u0441\u0456\u045e data<\/em><\/strong> \u043f\u0435\u0440\u0430\u0434\u0430\u044e\u0446\u0446\u0430 \u043d\u0430\u0441\u0442\u0443\u043f\u043d\u044b\u044f \u043f\u0430\u0440\u0430\u043c\u0435\u0442\u0440\u044b:<\/p>\n \u041f\u0440\u044b\u043a\u043b\u0430\u0434 AJAX-\u0437\u0430\u043f\u044b\u0442\u0443:<\/p>\n $.ajax({<\/span> \u0421\u0435\u0440\u0432\u0435\u0440 \u0432\u0435\u0440\u043d\u0435 JSON-\u043c\u0430\u0441\u0456\u045e \u0437 \u0443\u0432\u0430\u0445\u043e\u0434\u043d\u044b\u043c \u0442\u044d\u043a\u0441\u0442\u0430\u043c (\u043f\u0430\u0440\u0430\u043c\u0435\u0442\u0440 text<\/strong><\/em>), \u043c\u0430\u0441\u0456\u0432\u0430\u043c \u0441\u0430 \u0437\u0433\u0440\u0443\u043f\u0430\u0432\u0430\u043d\u044b\u043c\u0456 \u0441\u0456\u043c\u0432\u0430\u043b\u044c\u043d\u044b\u043c\u0456 \u043f\u0430\u0441\u043b\u044f\u0434\u043e\u045e\u043d\u0430\u0441\u0446\u044f\u043c\u0456 (\u043f\u0430\u0440\u0430\u043c\u0435\u0442\u0440 CharacterArr<\/strong><\/em>) \u0456 \u043c\u0430\u0441\u0456\u0432\u0430\u043c \u0437 \u0441\u0456\u043c\u0432\u0430\u043b\u044c\u043d\u044b\u043c\u0456 \u043f\u0430\u0441\u043b\u044f\u0434\u043e\u045e\u043d\u0430\u0441\u0446\u044f\u043c\u0456, \u0437\u0433\u0440\u0443\u043f\u0430\u0432\u0430\u043d\u044b\u043c\u0456 \u043f\u0430\u0432\u043e\u0434\u043b\u0435 \u0434\u0430\u0434\u0430\u0442\u043a\u043e\u0432\u044b\u0445 \u043f\u0440\u0430\u0432\u0456\u043b, (\u043f\u0430\u0440\u0430\u043c\u0435\u0442\u0440 ResultArr<\/strong><\/em>). \u041d\u0430\u043f\u0440\u044b\u043a\u043b\u0430\u0434, \u043f\u0430 \u0432\u044b\u0448\u044d\u0439 \u043f\u0440\u044b\u0432\u0435\u0434\u0437\u0435\u043d\u044b\u043c AJAX-\u0437\u0430\u043f\u044b\u0446\u0435 \u0431\u0443\u0434\u0437\u0435 \u0441\u0444\u0430\u0440\u043c\u0456\u0440\u0430\u0432\u0430\u043d\u044b \u043d\u0430\u0441\u0442\u0443\u043f\u043d\u044b \u0430\u0434\u043a\u0430\u0437:<\/p>\n [<\/span> Service \u00abTokenizer<\/a>\u00bb<\/strong> is intended to highlight tokens in the text. The text that requires tokenization is sent to the service input. After processing the text on the output, the user receives a list of the extended tokens.<\/p>\n <\/p>\n Basic terms and concepts<\/strong><\/p>\n Tokenization<\/em>\u00a0\u2014 selection of tokens in the text by a special computer program. This is the process of analyzing the input sequence of symbols into lexemes, in order to obtain the output of the identified sequences, the so-called \u00abtokens\u00bb. Lexical analysis is used in the compilers and interpreters of the source code of programming languages, and in various parsers of words of natural languages.<\/p>\n Token<\/em> (lexical analysis) \u2014 a sequence of characters in lexical analysis in computer science, corresponding to a lexeme. An object that is created from a lexeme in the process of lexical analysis (tokenization).<\/p>\n Token template<\/em>\u00a0\u2014 a formal description of the class of tokens that can create this type of token.<\/p>\n <\/p>\n User Interface Description<\/strong><\/p>\n The user interface of the service is shown in Figure 1.<\/p>\n <\/p>\n Figure 1.<\/strong>\u00a0User interface of the service \u00abTokenizer\u00bb<\/p>\n <\/p>\n The interface has the following areas:<\/strong><\/p>\n <\/p>\n User script for working with the service<\/strong><\/p>\n <\/p>\n <\/p>\n Access to the service via the API<\/strong><\/p>\n To access the service\u00a0\u00abTokenizer\u00bb<\/a><\/strong> via the API, you should send an AJAX-request (type: POST) to the address https:\/\/corpus.by\/Tokenizer\/api.php. With an input array\u00a0data\u00a0<\/strong><\/em>the following parameters are passed:<\/p>\n Example of AJAX-request:<\/p>\n $.ajax({<\/span> The server returns a JSON-array with the following parameters:<\/p>\n For example, the following reply will be formed on the above listed AJAX-request:<\/p>\n [<\/span> <\/p>\n Links to sources<\/strong><\/p>\n Service page:\u00a0https:\/\/corpus.by\/Tokenizer\/?lang=en<\/a><\/p>","protected":false},"excerpt":{"rendered":"\n
![\"\u041c\u0430\u043b\u044e\u043d\u0430\u043a](\"https:\/\/ssrlab.by\/wp-content\/uploads\/2017\/08\/Tokenizer_screen02.png\")
<\/a>\n
\n
\n\u00a0 \u00a0type: “POST”,<\/span>
\n\u00a0 \u00a0url: “https:\/\/corpus.by\/Tokenizer\/api.php”,<\/span>
\n\u00a0 \u00a0data:{<\/span>
\n\u00a0 \u00a0 \u00a0 “inputText”: “\u0421\u0456\u0301\u0440\u044b\u0443\u0441 (\u043b\u0430\u0446. Sirius), \u0442\u0430\u043a\u0441\u0430\u043c\u0430 \u0410\u043b\u044c\u0444\u0430 \u0412\u044f\u043b\u0456\u043a\u0430\u0433\u0430 \u0421\u0430\u0431\u0430\u043a\u0456 (\u043b\u0430\u0446. Alpha Canis Majoris) \u2013 \u043d\u0430\u0439\u044f\u0440\u0447\u044d\u0439\u0448\u0430\u044f \u0437\u043e\u0440\u043a\u0430 \u043d\u0430\u0447\u043d\u043e\u0433\u043e \u043d\u0435\u0431\u0430.”,<\/span>
\n\u00a0 \u00a0 \u00a0 “language”: “be”<\/span>
\n\u00a0 \u00a0},<\/span>
\n\u00a0 \u00a0success: function(msg){ },<\/span>
\n\u00a0 \u00a0error: function() { }<\/span>
\n});<\/span><\/span><\/p><\/blockquote>\n
\n\u00a0 \u00a0{<\/span>
\n\u00a0 \u00a0 \u00a0 “text”: “\u0421\u0456\u0301\u0440\u044b\u0443\u0441 (\u043b\u0430\u0446. Sirius), \u0442\u0430\u043a\u0441\u0430\u043c\u0430 \u0410\u043b\u044c\u0444\u0430 \u0412\u044f\u043b\u0456\u043a\u0430\u0433\u0430 \u0421\u0430\u0431\u0430\u043a\u0456 (\u043b\u0430\u0446. Alpha Canis Majoris) \u2013 \u043d\u0430\u0439\u044f\u0440\u0447\u044d\u0439\u0448\u0430\u044f \u0437\u043e\u0440\u043a\u0430 \u043d\u0430\u0447\u043d\u043e\u0433\u043e \u043d\u0435\u0431\u0430.”,<\/span>
\n\u00a0 \u00a0 \u00a0 “CharacterArr”:\u00a0<\/span>
\n\u00a0 \u00a0 \u00a0 {<\/span>
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0“0”: {<\/span>
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 “0”:\u00a0“\u0421\u0456”,<\/span>
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 “1”:\u00a0“alphabet”,<\/span>
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 “2”:\u00a02<\/span>
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0},<\/span>
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0“1”: {<\/span>
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 “0”: “\u0301”,<\/span>
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 “1”:\u00a0“character”,<\/span>
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 “2”:\u00a01<\/span>
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0},<\/span>
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0“2”: {<\/span>
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 “0”: “\u0440\u044b\u0443\u0441”,<\/span>
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 “1”:\u00a0“alphabet”,<\/span>
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 “2”:\u00a04<\/span>
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0},<\/span>
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0<…><\/span>
\n\u00a0 \u00a0 \u00a0 },<\/span>
\n\u00a0 \u00a0 \u00a0 “ResultArr”:\u00a0{<\/span>
\n\u00a0 \u00a0\u00a0 \u00a0\u00a0 \u00a0“0”: {<\/span>
\n\u00a0 \u00a0\u00a0 \u00a0\u00a0 \u00a0\u00a0 \u00a0“0”:\u00a0“\u0421\u0456\u0301\u0440\u044b\u0443\u0441”,<\/span>
\n\u00a0 \u00a0\u00a0 \u00a0\u00a0 \u00a0\u00a0 \u00a0“1”:\u00a0“word”,<\/span>
\n\u00a0 \u00a0\u00a0 \u00a0\u00a0 \u00a0 \u00a0\u00a0“2”:\u00a07<\/span>
\n\u00a0 \u00a0\u00a0 \u00a0\u00a0 \u00a0},<\/span>
\n\u00a0 \u00a0\u00a0 \u00a0\u00a0 \u00a0“1”: {<\/span>
\n\u00a0 \u00a0\u00a0 \u00a0 \u00a0\u00a0\u00a0 \u00a0“0”: ” “,<\/span>
\n\u00a0 \u00a0\u00a0 \u00a0 \u00a0\u00a0\u00a0 \u00a0“1”:\u00a0“whitespaces”,<\/span>
\n\u00a0 \u00a0\u00a0 \u00a0\u00a0 \u00a0\u00a0 \u00a0“2”:\u00a01<\/span>
\n\u00a0 \u00a0\u00a0\u00a0 \u00a0 \u00a0},<\/span>
\n\u00a0 \u00a0\u00a0 \u00a0 \u00a0\u00a0“2”: {<\/span>
\n\u00a0 \u00a0\u00a0 \u00a0 \u00a0\u00a0\u00a0 \u00a0“0”: “(“,<\/span>
\n\u00a0 \u00a0\u00a0\u00a0 \u00a0 \u00a0\u00a0 \u00a0“1”:\u00a0“character”,<\/span>
\n\u00a0 \u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0 \u00a0“2”: 1<\/span>
\n\u00a0 \u00a0\u00a0 \u00a0 \u00a0\u00a0},<\/span>
\n\u00a0 \u00a0\u00a0\u00a0 \u00a0 \u00a0<…><\/span>
\n\u00a0 \u00a0 \u00a0\u00a0}<\/span>
\n\u00a0 \u00a0}<\/span>
\n]<\/span><\/p><\/blockquote>\n<\/a><\/p>\n\n
\n
<\/a>Figure 2.<\/strong>\u00a0\u00abTokenizer\u00bb results example<\/p>\n\n
\n\u00a0 \u00a0type: “POST”,<\/span>
\n\u00a0 \u00a0url: “https:\/\/corpus.by\/Tokenizer\/api.php”,<\/span>
\n\u00a0 \u00a0data:{<\/span>
\n\u00a0 \u00a0 \u00a0 “inputText”: “\u0421\u0456\u0301\u0440\u044b\u0443\u0441 (\u043b\u0430\u0446. Sirius), \u0442\u0430\u043a\u0441\u0430\u043c\u0430 \u0410\u043b\u044c\u0444\u0430 \u0412\u044f\u043b\u0456\u043a\u0430\u0433\u0430 \u0421\u0430\u0431\u0430\u043a\u0456 (\u043b\u0430\u0446. Alpha Canis Majoris) \u2013 \u043d\u0430\u0439\u044f\u0440\u0447\u044d\u0439\u0448\u0430\u044f \u0437\u043e\u0440\u043a\u0430 \u043d\u0430\u0447\u043d\u043e\u0433\u043e \u043d\u0435\u0431\u0430.”,<\/span>
\n\u00a0 \u00a0 \u00a0 “language”: “be”<\/span>
\n},
\nsuccess: function(msg){ },
\nerror: function() { }
\n});<\/span><\/p><\/blockquote>\n\n
\n\u00a0 \u00a0{<\/span>
\n\u00a0 \u00a0 \u00a0 “text”: “\u0421\u0456\u0301\u0440\u044b\u0443\u0441 (\u043b\u0430\u0446. Sirius), \u0442\u0430\u043a\u0441\u0430\u043c\u0430 \u0410\u043b\u044c\u0444\u0430 \u0412\u044f\u043b\u0456\u043a\u0430\u0433\u0430 \u0421\u0430\u0431\u0430\u043a\u0456 (\u043b\u0430\u0446. Alpha Canis Majoris) \u2013 \u043d\u0430\u0439\u044f\u0440\u0447\u044d\u0439\u0448\u0430\u044f \u0437\u043e\u0440\u043a\u0430 \u043d\u0430\u0447\u043d\u043e\u0433\u043e \u043d\u0435\u0431\u0430.”,<\/span>
\n\u00a0 \u00a0 \u00a0 “CharacterArr”: <\/span>{<\/span>
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0“0”: {<\/span>
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 “0”:\u00a0“\u0421\u0456”,<\/span>
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 “1”:\u00a0“alphabet”,<\/span>
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 “2”:\u00a02<\/span>
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0},<\/span>
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0“1”: {<\/span>
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 “0”: “\u0301”,<\/span>
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 “1”:\u00a0“character”,<\/span>
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 “2”:\u00a01<\/span>
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0},<\/span>
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0“2”: {<\/span>
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 “0”: “\u0440\u044b\u0443\u0441”,<\/span>
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 “1”:\u00a0“alphabet”,<\/span>
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 “2”:\u00a04<\/span>
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0},<\/span>
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0<…><\/span>
\n\u00a0 \u00a0 \u00a0 },<\/span>
\n\u00a0 \u00a0 \u00a0 “ResultArr”:\u00a0{<\/span>
\n\u00a0 \u00a0\u00a0 \u00a0\u00a0 \u00a0“0”: {<\/span>
\n\u00a0 \u00a0\u00a0 \u00a0\u00a0 \u00a0\u00a0 \u00a0“0”:\u00a0“\u0421\u0456\u0301\u0440\u044b\u0443\u0441”,<\/span>
\n\u00a0 \u00a0\u00a0 \u00a0\u00a0 \u00a0\u00a0 \u00a0“1”:\u00a0“word”,<\/span>
\n\u00a0 \u00a0\u00a0 \u00a0\u00a0 \u00a0 \u00a0\u00a0“2”:\u00a07<\/span>
\n\u00a0 \u00a0\u00a0 \u00a0\u00a0 \u00a0},<\/span>
\n\u00a0 \u00a0\u00a0 \u00a0\u00a0 \u00a0“1”: {<\/span>
\n\u00a0 \u00a0\u00a0 \u00a0 \u00a0\u00a0\u00a0 \u00a0“0”: ” “,<\/span>
\n\u00a0 \u00a0\u00a0 \u00a0 \u00a0\u00a0\u00a0 \u00a0“1”:\u00a0“whitespaces”,<\/span>
\n\u00a0 \u00a0\u00a0 \u00a0\u00a0 \u00a0\u00a0 \u00a0“2”:\u00a01<\/span>
\n\u00a0 \u00a0\u00a0\u00a0 \u00a0 \u00a0},<\/span>
\n\u00a0 \u00a0\u00a0 \u00a0 \u00a0\u00a0“2”: {<\/span>
\n\u00a0 \u00a0\u00a0 \u00a0 \u00a0\u00a0\u00a0 \u00a0“0”: “(“,<\/span>
\n\u00a0 \u00a0\u00a0\u00a0 \u00a0 \u00a0\u00a0 \u00a0“1”:\u00a0“character”,<\/span>
\n\u00a0 \u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0 \u00a0“2”: 1<\/span>
\n\u00a0 \u00a0\u00a0 \u00a0 \u00a0\u00a0},<\/span>
\n\u00a0 \u00a0\u00a0\u00a0 \u00a0 \u00a0<…><\/span>
\n\u00a0 \u00a0 \u00a0\u00a0}<\/span>
\n\u00a0 \u00a0}<\/span>
\n]<\/span><\/p><\/blockquote>\n