Jump to content

Tokenization (lexical analysis)

From Wikipedia, the free encyclopedia

This is the current revision of this page, as edited by Qwerfjkl (talk | contribs) at 03:46, 2 February 2023 (Converting {{Wikidata redirect}} to {{R with Wikidata item}}. {{Wikidata redirect}} should only be used on soft redirects.). The present address (URL) is a permanent link to this version.

(diff) ← Previous revision | Latest revision (diff) | Newer revision → (diff)

Redirect to:

Lexical analysis#Tokenization

This page is a redirect. The following categories are used to track and monitor this redirect:

From a merge: This is a redirect from a page that was merged into another page. This redirect was kept in order to preserve the edit history of this page after its content was merged into the content of the target page. Please do not remove the tag that generates this text (unless the need to recreate content on this page has been demonstrated) or delete this page.
- For redirects with substantive page histories that did not result from page merges use {{R with history}} instead.

To a section: This is a redirect from a topic that does not have its own page to a section of a page on the subject. For redirects to embedded anchors on a page, use {{R to anchor}} instead.

From a cross-project redirect: This is a redirect from a title associated with an item on Wikidata. The Wikidata item associated with this page is tokenization (Q2438971).
- Use this template only on hard redirects – for soft redirects use {{Soft redirect with Wikidata item}}.

When appropriate, protection levels are automatically sensed, described and categorized.

Retrieved from "https://en.wikipedia.org/w/index.php?title=Tokenization_(lexical_analysis)&oldid=1136980029"

Hidden categories: