Jump to content

Tokenization (lexical analysis)

From Wikipedia, the free encyclopedia

This is an old revision of this page, as edited by Neo-Jay (talk | contribs) at 14:00, 22 February 2020 (added Template:Wikidata redirect (Wikidata:Q2438971)). The present address (URL) is a permanent link to this revision, which may differ significantly from the current revision.

(diff) ← Previous revision | Latest revision (diff) | Newer revision → (diff)

Redirect page

Redirect to:

Lexical analysis#Tokenization

This page is a redirect. The following categories are used to track and monitor this redirect:

From a merge: This is a redirect from a page that was merged into another page. This redirect was kept in order to preserve the edit history of this page after its content was merged into the content of the target page. Please do not remove the tag that generates this text (unless the need to recreate content on this page has been demonstrated) or delete this page.
- For redirects with substantive page histories that did not result from page merges use {{R with history}} instead.

To a section: This is a redirect from a topic that does not have its own page to a section of a page on the subject. For redirects to embedded anchors on a page, use {{R to anchor}} instead.

From a cross-project redirect: This is a soft redirect that is used as a connection to other Wikimedia projects. A Wikidata element is linked to this page:

tokenization (Q2438971).

Use this template only on soft redirects – for hard redirects use {{R with Wikidata item}}.

When appropriate, protection levels are automatically sensed, described and categorized.

Retrieved from "https://en.wikipedia.org/w/index.php?title=Tokenization_(lexical_analysis)&oldid=942083152"

Hidden categories: