View source for Help:Searching/Regex/Sandboxing

=== Developing regular expressions in ''an ad'' hoc sandbox ===

[[Regular expressions]] are little computer programs, so it is characteristic of regex searches that they must be written while studying the target data, and tested to achieve their potential precision and thoroughness. However, only a few of these intensive searches are technically able to run at a time against the database.<ref>See [https://discuss.elastic.co/t/how-to-protect-an-es-cluster-from-searches-that-would-kill-it/25487 how searches can  kill Wikipedia's CirrusSearch/ElasticSearch engine]</ref> A sandbox minimizes your footprint, and guarantees that you will never run an untested regexp on every namespace in the wiki, even if [[H:S#User preferences|your default search]] would let you do that.

Although a ''normal'' search targeting the entire wiki will run quickly, a ''regexp'' search should target as few pages as possible by using filters in order to run quickly. A filter is part or whole of a ''database query''. Filters include:
* word(s) or phrase
* intitle:
* incategory:
* hastemplate:
* prefix: (always at the end)
* linksto:
* ''namespace:'' (always at the beginning)
* insource:"word1 word2"
* insource:word

Order is not important because the search is optimized by the software before it is run.

To target just one page while experimenting with or developing a regex search, target a fullpagename. From the search box use the filter <kbd>prefix:''fullpagename''</kbd>. From the edit box (of any section of the page with the target data), you can always just write <kbd>{{nowrap|prefix:{<nowiki/>{FULLPAGENAME}}}}</kbd> and it will "expand" for you to the fullpagename. Although you can edit a history page, technically a "history page" is not a page (in the database), and so {<nowiki/>{FULLPAGENAME}} ''there'' will point to the database version (not its own rendering). For the same reason, you cannot search for the wikitext on a page that is not already saved (to the database), although you can certainly change the search parameters again and again with no need to save ''them''.

''Fullpagename is namespace:pagename''.  Knowing this you can adjust your Prefix parameter.  Although ''prefix'' can filter down to one page, it can filter up to a namespace, and it also accepts the beginning letter(s) of set of pagenames if you want to reduce the namespace search domain.

Regex sandboxing uses an ''ad hoc'' sandbox made by editing any page containing the target data, and using it as a "sandbox" (not editing it to save it). It then develops by using adding a search link that includes insource:/''regexp''/, with the filter ''prefix:&#123;{FULLPAGENAME}}'' alongside.

Use of a sandbox enables the smallest possible footprint by using filters to limit the search domain. Once your regexp pattern is honed, you increase the search domain. A regex search is best run with filters, not alone even if it is a polished rexexp.

==== Sandboxing procedure ====

Rather than use the search box, where entering an equals sign and a pipe character, and "quotes around phrases" is a straightforward matter, it is still easiest to use a regex-based search-link template {{mdash}} {{tl|regex}} or {{tl|tlusage}} {{mdash}} on the page with sample data, because then you can focus on the target data there and on writing the regexp pattern. It is easier, that is, if you already understand how templates "escape" the pipe character and the equals sign. See [[Help:Template#Parameters]] for other important details.

The procedure here is an iterative, read-evaluate-modify cycle. Regex development requires that you study the target data while writing and rewriting its pattern.
# Navigate to a page with the wikitext instances you are interested in mining. Or create one yourself, and save it to the database so the query will find it.  
# Open the wikitext, and enter a {{tl|regex}} or {{tl|tlusage}}.
# ''Show preview'', and activate the search link. On the search results page, note the bold text in each match.
# Go back in your browser. Modify the regexp, and cycle until done. (Or don't go back, you may want to modify the query at the search box.)
# Expand the search domain, and test the accuracy of ''those'' results.  You can trim or expand the number of the results using ''prefix:''.

''Caveat emptor'': if you change the ''target'' for an ''immediate'' retesting, you'll have to save and [[WP:Purge|purge]], but not if you just change the regexp.
<section begin=Examples />

==== Examples ====

As an ''ad hoc'' sandbox, you can show the wikitext of a section like this, (already saved in the database), modify some of the patterns in the regex-search-link template calls on this page, do a Show Preview, and see what matches when you click on the newly formed regex search-link, all quite safely, and without changing a thing in the database.

<!--If you change the below, sample, targets, you'll have to save the page
to get them into the database to find them in the query. The first batch here
is concentrated so that it shows up to six or seven hits on the results page.
vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv -->
The template calls that produce "{{val|1|ul=ft/s|fmt =   commas}},
{{val|2|u=ft2}},
{{val|3|u=m/s| fmt =commas }},
{{val|4|u=m*s-2}},
{{val|5|u=ft.s-2}},
{{val|6|u=C/J}}, and
{{val|7|ul=J/C}}"
appear in the wikitext of this section like this:
# <kbd><nowiki>{{val|1|ul=ft/s|fmt =   commas}}</nowiki></kbd> 
# <kbd><nowiki>{{val|2|u=ft2}}</nowiki></kbd> 
# <kbd><nowiki>{{val|3|u=m/s| fmt =commas }}</nowiki></kbd> 
# <kbd><nowiki>{{val|4|u=m*s-2}}</nowiki></kbd> 
# <kbd><nowiki>{{val|5|u=ft.s-2}}</nowiki></kbd>
# <kbd><nowiki>{{val|6|u=C/J}}</nowiki></kbd> 
# <kbd><nowiki>{{val|7|ul=J/C}}</nowiki></kbd>
<!--^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If you change the above targets, you'll have to save the page
to get them into the database to find them in the query.-->

Note how the above targets are |numbered|, then click on the links below.

{| class="wikitable"
! scope="col" width="200px" | Query
! [[Template:Internal search templates common doc#Search links|Search link]]
! Answer
|-
|'''Q1''' Using {{tl|search link}}, does this page ''employ'' template Val ?
|{{#invoke:DemoTemplate|sl|hastemplate: Val}} 
|'''A.''' No, because this pagename is in Help not Article space.(Search link default). '''1300''' search results. 
|-
|'''Q2''' Using {{tlf|search link}} ''responsibly'', does this page use Val's fmt parameter?
|<code><nowiki>{{sl|insource:/\{[Vv]al\{{!}}[^}]*fmt/ prefix:{{FULLPAGENAME}}}}</nowiki></code> &rarr; 
{{sl|insource:/\{[Vv]al\{{!}}[^}]*fmt/ prefix:{{FULLPAGENAME}}}} 
|'''A2.1'''. Look for 1 and 3 in the search results in bold text. (Adds an appropriate filter.)
|-
|Using {{tl|regex}} instead...
|<code><nowiki>{{slre|\{[Vv]al\{{!}}[^}]*fmt}}</nowiki></code> &rarr;
{{slre|\{[Vv]al\{{!}}[^}]*fmt}}
|'''A2.2''' Less typing than {{tlf|search link}}.
|-
|Using {{tl|template usage}} instead...
|<code><nowiki>{{tlre|Val|pattern=fmt}}</nowiki></code> &rarr; 
{{tlre|Val|pattern = fmt}}
|'''A2.3''' Easiest for templates.
|-

|'''Q3'''. Who uses <kbd>u=ft</kbd> OR <kbd>ul=ft</kbd>? (one-letter differs)
| <code><nowiki>{{regex|ul?=ft}}</nowiki></code> &rarr;
{{slre|ul?{{=}}ft}}
|'''A'''. Look for 1, 2, and 5 in bold text. 
|-
|Using {{tl|template usage}}...
|<code><nowiki>{{tlre|val|pattern = ul?=ft}}</nowiki></code> &rarr;
{{tlre|val|pattern = ul?=ft}}
|Finds same pattern, but only ''inside'' a Val template.

|-
|'''Q4'''. AND of these, who also uses fmt=commas after that?
| <code><nowiki>{{slre|ul?=ft.*commas}}</nowiki></code> &rarr;
{{slre|ul?=ft.*commas}}
|A. No context shown, but article title is shown. A half a Bug?
|-
|Who has one space before the word "commas"?
|{{#invoke:DemoTemplate|slre|. commas}}
|'''A'''. 1 but not 2.

|-
|'''Q5'''. Who uses either u or ul with "ft" OR uses "fmt=commas". 
|<code><nowiki>{{slre|(ul? *= *ft{{!}}fmt *= *commas)}}</nowiki></code>
&rarr; {{slre|(ul? *= *ft{{!}}fmt *= *commas)}}
|'''A'''. 1, 2, 3, and 5. (The pattern matches all possible spacing.)

|-
|'''Q6'''. Who uses ''ft'' or ''m'', in {{para|u}} or {{para|ul}}?
|<code><nowiki>{{slre|ul? *{{=}} *(ft{{!}}m)}}</nowiki></code> 
&rarr; {{slre|ul? *{{=}} *(ft{{!}}m)}}
|'''A'''. 1, 2, 3, 4, and 5.  
Used {<nowiki/>{!}} for the ''alternation'' metacharacter. Used {{tl|1==}}. (Could have used named <code>1 = </code> or nicely named <code>pattern = </code>.)

|-
|'''Q7'''. Who uses . or * in the unit code?
|<code><nowiki>{{tlre|val|pattern = u *= *(\.{{!}}\*)/}}</nowiki></code>
&rarr; {{tlre|val|pattern = u *= *(\.{{!}}\*)/}}
|'''A'''. 4 and 5.
|-
|Who uses a pipe?
|{{#invoke:DemoTemplate|regex|\{{!}}}}
|All of them

|-
|'''Q8'''. Who uses / or - ''within'' the {{para|u}} or {{para|ul}} parameter?
|<code><nowiki>{{tlre|val|ul? *= *[^{{!}}}]+(\/{{!}}-)}}</nowiki></code> 
&rarr; {{tlre|val|pattern = ul? *= *[^{{!}}}]+(\/{{!}}-)}}
|'''A'''. 1,3,4,5,6 and 7.

|-
|'''Q9'''. Where is Val used in the template namespace for numbers only, (no <kbd>u</kbd>, <kbd>ul</kbd>, <kbd>up</kbd>, or <kbd>upl</kbd> parameters).
|<code><nowiki>{{tlre|val|pattern = ~(u[lp].)|prefix = 10}}</nowiki></code>
&rarr; {{tlre|val|pattern = ~(u[lp].)|prefix = 10 }}
|'''A'''. In the 30 or so templates listed.

|-
| '''Q10'''. Which articles use {{tl|Convert}}'s <kbd>and(-)</kbd> option?
|<code><nowiki>{{tlre|convert|pattern=and\(-\)| prefix=0}}</nowiki></code>
&rarr; {{tlre|convert|pattern = and\(-\) | prefix = 0}}
| '''A''' ''Coast Range Arc'' and ''Skipjack shad''
|}

In '''Q2''', notice how the MediaWiki software ignores the spaces ''around'' parameters, but how in '''Q4''' the same MediaWiki software processes the spaces ''inside'' parameters. Q2 might have been solved with a plain <kbd>insource:val fmt</kbd> search because "fmt" and "val" are whole words, and fmt is rarely seen apart from inside Val.  How about <kbd>hastemplate:val insource:fmt</kbd>?
<section end=Examples />

==References==
{{Reflist}}