View source for Module talk:DecodeEncode

{{Permanently protected}}

==Bug report: bad decoding of U+03B5 ε (epsilon) <span class="anchor" id="U+03B5"></span><span class="anchor" id="epsilon"></span>==
{{tracked|T328840}}
About <span class="nowrap"><templatestyles src="Mono/styles.css" /><span class="monospaced">U+03B5</span>&#x20;</span><span style="font-size:125%;line-height:1em">&#x03B5;</span> <span style="font-size:85%;">GREEK SMALL LETTER EPSILON</span> (<templatestyles src="Mono/styles.css" /><span class="monospaced">&amp;epsi; &amp;epsilon;</span>)

* '''Issue''': after resolving HTML entity <code>&amp;epsilon;</code> by <code>[[:mw:Extension:Scribunto/Lua_reference_manual#mw.text.decode|mw.text.decode()]]</code>, the plain character is ''not found'' by <code>[[:mw:Extension:Scribunto/Lua_reference_manual#mw.ustring.gsub|mw.ustring.gsub()]]</code>. No issue with alternative HTML entity <code>&amp;epsi;</code>. {{xt|&amp;epsi;}} good, {{!xt|&amp;epsilon;}} bad.

:Report limitations: Original report and bug reproduction is at enwiki [[:en:Module talk:DecodeEncode#epsilon"|Module talk:DecodeEncode]], from where [[:en:module:DecodeEncode]] and [[:en:module:String]] are used live. At phabricator pseudocode may be used and some "results" may be hardcoded.  In-text the escape <code>&amp;amp;</code> is used, not in-function. Lua patterns not used ("no <code>%</code>").

*To reproduce:
:1. Create research string:
::<code>X&amp;epsi;1X&amp;epsilon;2X</code> (shows live and unedited as: X&epsi;1X&epsilon;2X)<!-- Xε1Xε2X -->
:2. Render the string by <code>decode()</code> (as inner function)
:3. then on rendered result use <code>gsub()</code> to replace plain character <code>ε</code> &rarr; <code>E</code>: (as outer function)
::<code>mw.ustring.gsub( s=(<code>mw.text.decode( s=X&amp;epsi;1X&amp;epsilon;2X, decodeNamedEntities=true )</code> ), pattern=ε, repl=E )</code> [is pseudo-code, see note. 21:10, 7 February 2023 (UTC)]
:4. Result3 (s&r pattern use ε from <code>Xε1X</code>):
::{{#invoke:String|replace|source={{#invoke:DecodeEncode|decode|s=X&epsi;1X&epsilon;2X}}|pattern=ε|replace=E|plain=true}}<!-- from Xε1X -->
:5. Result4 (s&r pattern use ε from <code>Xε2X</code>):
::{{#invoke:String|replace|source={{#invoke:DecodeEncode|decode|s=X&epsi;1X&epsilon;2X}}|pattern=ε|replace=E|plain=true}}<!-- from Xε2X -->
* Expected: <code>XE1XE2X</code> (only one character <code>ε</code> exists)

* ''Note 21:10, 7 February 2023 (UTC):'' This step 3 is in pseudo-code. To reproduce, use Lua modules [[:module:String]] and [[:Module:DecodeEncode]]:
::<!-- in THIS line: using "&amp;" escape (so copy/paste works). In the NEXT (live) line: using "s=X&epsi;1X&epsilon;2X" [literal] --><code><nowiki>{{#invoke:String|replace|source={{#invoke:DecodeEncode|decode|s=X&amp;epsi;1X&amp;epsilon;2X}}|pattern=ε|replace=E|plain=true}}</nowiki></code>
::&rarr; {{#invoke:String|replace|source={{#invoke:DecodeEncode|decode|s=X&epsi;1X&epsilon;2X}}|pattern=ε|replace=E|plain=true}}
::''-[[User:DePiep|DePiep]] ([[User talk:DePiep|talk]]) 21:10, 7 February 2023 (UTC)''
===Workaround A, ad hoc===
Workaround A, ad hoc: add innermost function to ''first'' replace in the research string <code>&amp;epsilon;</code> &rarr; <code>&amp;epsi;</code>: 
:A1: <code><nowiki>{{#invoke:String|replace|source={{#invoke:DecodeEncode|decode|s={{#invoke:String|replace|source=X&amp;epsi;1X&amp;epsilon;2X|pattern=&amp;epsilon;|replace=&amp;epsi;|plain=true}}}}|pattern=ε|replace=E|plain=true}}</nowiki></code> &rarr;
:{{#invoke:String|replace|source={{#invoke:DecodeEncode|decode|s={{#invoke:String|replace|source=X&epsi;1X&epsilon;2X|pattern=&epsilon;|replace=&epsi;|plain=true}}}}|pattern=ε|replace=E|plain=true}}<!-- from Xε1X -->

=== Workaround B, in module (THIN SPACE example) <span class="anchor" id="Workaround B"></span>===
Workaround B: early in [[:en:module:DecodeEncode#Decode|:en:module:DecodeEncode]], replace <code>&amp;epsilon;</code> &rarr; <code>&amp;epsi;</code> 

About THIN SPACE: it looks like character <span class="nowrap"><templatestyles src="Mono/styles.css" /><span class="monospaced">U+2009</span>&#x20;</span><span style="font-size:85%;">THIN SPACE</span> (<templatestyles src="Mono/styles.css" /><span class="monospaced">&amp;thinsp; &amp;ThinSpace;</span>) has a samilar issue. {{xt|&amp;ThinSpace;}} good, {{!xt|&amp;thinsp;}} bad.

Currently in code:
<syntaxhighlight lang="Lua" line start="33" highlight="3">
function p._decode( s, subset_only )
	local ret = nil;
    s = mw.ustring.gsub( s, '&thinsp;', '&ThinSpace;' ) -- Workaround for bug: &ThinSpace; gets properly decoded in decode, but &thinsp; doesn't.
	ret = mw.text.decode( s, not subset_only )
	return ret
end
</syntaxhighlight>

In [[:en:module:DecodeEncode/sandbox]], I have coded a similar handling of EPSILON:
:[[:module:DecodeEncode]], [[:module:DecodeEncode/sandbox]] {{diffsandbox|Module:DecodeEncode}}

<syntaxhighlight lang="Lua" line start="33" highlight="5-6">
function p._decode( s, subset_only )
	local ret = nil;
	-- U+2009 THIN SPACE: workaround for bug: HTML entity &thinsp; is decoded incorrect. Entity &ThinSpace; gets decoded properly
	s = mw.ustring.gsub( s, '&thinsp;', '&ThinSpace;' )
	-- U+03B5 ε GREEK SMALL LETTER EPSILON: workaround for bug (phab:T328840): HTML entity &epsilon; is decoded incorrect for gsub(). Entity &epsi; gets decoded properly
	s = mw.ustring.gsub( s, '&epsilon;', '&epsi;' )
	ret = mw.text.decode( s, not subset_only )
	return ret
end</syntaxhighlight>

* /sandbox tests:
:B. <code><nowiki>{{#invoke:String|replace|source={{#invoke:DecodeEncode/sandbox|decode|s=X&amp;epsi;1X&amp;epsilon;2X}}|pattern=ε|replace=E|plain=true}}</nowiki></code>
:B1. ResultB1 (s&r pattern use ε from <code>Xε1X</code>): {{#invoke:String|replace|source={{#invoke:DecodeEncode/sandbox|decode|s=X&epsi;1X&epsilon;2X}}|pattern=ε|replace=E|plain=true}}<!-- from Xε1X -->
:B2. ResultB2 (s&r pattern use ε from <code>Xε2X</code>): {{#invoke:String|replace|source={{#invoke:DecodeEncode/sandbox|decode|s=X&epsi;1X&epsilon;2X}}|pattern=ε|replace=E|plain=true}}<!-- from Xε2X -->

I propose to edit the module along this way.

===Workaround C (mw, Lua)===
Changes in mw, Lua: I have not idea.

* I propose to consider module editing along {{slink||Workaround B}}. -[[User:DePiep|DePiep]] ([[User talk:DePiep|talk]]) 12:26, 4 February 2023 (UTC)

===testcases EPSILON===
;* ''Original failure, now solved=not showing any more<nowiki>:</nowiki>''
::(hardcoded explanation here): in cell marked {{nay}}, the result showed as "XE1X&epsilon;2X". That is: wikitext input "{{code|1=&epsilon;}}" was ''not'' recognised & replaced. -[[User:DePiep|DePiep]] ([[User talk:DePiep|talk]]) 07:49, 19 February 2023 (UTC)

{| class="wikitable" style="font-size:100%"
|-
! colspan="6" | EPSILON <big>ε</big> {{angbr|1=<code>&amp;epsi;</code>}} error & fix proposal {{nobold|1=(16 Feb 2023)}}
|-
!1
!2
!3
!4
!5
!6
|-
! id
! entity code
! plain
! [[:Module:DecodeEncode|mod:.. decode(&amp;entity;)]] 
! replace(decode(..)) with E<br/> <small>pattern=hardcoded {{angbr|1=ε}} from plain</small><!--
--><br/>(s=&amp;entity;)<br/>(s=checkstring)
! [[:Module:DecodeEncode/sandbox|mod:..decode/sandbox]]
|-
| checkstring 
| {{code|1=X&epsi;1X&epsilon;2X}} 
| <big>>X&epsi;1X&epsilon;2X<</big>
| <big>>{{#invoke:DecodeEncode|decode|s=X&epsi;1X&epsilon;2X}}<</big>
|
|
|-
| EPSI
| {{code|&epsi;}}
| <big>>&epsi;<</big> 
| <big>>{{#invoke:DecodeEncode|decode|s=&epsi;}}<</big>
|{{#invoke:String|replace|source={{#invoke:DecodeEncode|decode
 |s=&epsi;}}|pattern=ε|replace=E|plain=true}}<br/>{{#invoke:String|replace|source={{#invoke:DecodeEncode|decode
 |s=X&epsi;1X&epsilon;2X}}|pattern=ε|replace=E|plain=true}}
| <!-- DecodeEncode/sandbox -->{{#invoke:String|replace|source={{#invoke:DecodeEncode/sandbox|decode
 |s=&epsi;}}|pattern=ε|replace=E|plain=true}}<br/>{{#invoke:String|replace|source={{#invoke:DecodeEncode/sandbox|decode
 |s=X&epsi;1X&epsilon;2X}}|pattern=ε|replace=E|plain=true}}
|-
| EPSILON
| {{code|&epsilon;}}
| <big>>&epsilon;<</big> 
| <big>>{{#invoke:DecodeEncode|decode|s=&epsilon;}}<</big>
| {{#invoke:String|replace|source={{#invoke:DecodeEncode|decode
 |s=&epsilon;}}|pattern=ε|replace=E|plain=true}}<br/>{{#invoke:String|replace|source={{#invoke:DecodeEncode|decode
 |s=X&epsi;1X&epsilon;2X}}|pattern=ε|replace=E|plain=true}}<br/>{{nay}}
| <!-- DecodeEncode/sandbox -->{{#invoke:String|replace|source={{#invoke:DecodeEncode/sandbox|decode
 |s=&epsilon;}}|pattern=ε|replace=E|plain=true}}<br/>{{#invoke:String|replace|source={{#invoke:DecodeEncode/sandbox|decode
 |s=X&epsi;1X&epsilon;2X}}|pattern=ε|replace=E|plain=true}}
|}

* See {{slink|Module_talk:DecodeEncode|Workaround_B,_in_module_(THIN_SPACE_example)|nopage=yes}} for code change;
:Similar fix as {{unichar|2009|THIN SPACE|html=}} has (though original cause bug may be different for THIN SPACE).
* Phabricator {{phab|T328840}} did not gain traction. Would be mw-level, not this module.
:-[[User:DePiep|DePiep]] ([[User talk:DePiep|talk]]) 06:22, 16 February 2023 (UTC)

=== Template-protected edit request on 16 February 2023 ===

{{edit template-protected|Module:DecodeEncode|answered=yes}}

* '''Please copy all code''' from [[:module:DecodeEncode/sandbox]] into [[:module:DecodeEncode]] ({{diffsandbox|Module:DecodeEncode}})
:''Issue:'' bad decoding of HTML entity <code>&amp;epsi;</code> {{nay}}
:re {{unichar|03B5|GREEK SMALL LETTER EPSILON|html=}}
:''Change:'' fix by replacing with entity <code>&amp;epsilon;</code> {{aye}} before applying <code>decode()</code>. See {{slink||Workaround B}} for code diff & backgrounds; minor comment change
:''Discussion:'' (1) reported at {{phab|T328840}}, no responses (mw-level); (2) [[#U+03B5|bug report]] here not challenged
:''Testcases:'' See {{slink||testcases EPSILON}}.
:[[User:DePiep|DePiep]] ([[User talk:DePiep|talk]]) 06:49, 16 February 2023 (UTC)
:: {{done}} [[User:Pppery|* Pppery *]] [[User talk:Pppery|<sub style="color:#800000">it has begun...</sub>]] 03:11, 19 February 2023 (UTC)

== NBSP behaviour ==

Leaving this note here.

About [[NBSP]], {{unichar|00a0|NO-BREAK SPACE|html=}}. With input {{code|1=&nbsp;}} I am experiencing problems reminding of {{slink||epsilon}} ({{phab|T328840}}, now resolved).

When nested like: (replace|s=(decode|s={{code|1=AB&nbsp;YZ}})|replace=AB_YZ) ''returns breaking code ''(breaking when used in/with HTML/css code like span, sup, class).

No time to build the reproduction/test, so have to leave it for now. Not reported on phab. [[User:DePiep|DePiep]] ([[User talk:DePiep|talk]]) 07:27, 20 February 2023 (UTC)

== Template-protected edit request on 21 March 2023 ==

{{edit template-protected|Module:DecodeEncode|answered=yes}}
Please replace all code [[:Module:DecodeEncode]] with [[:module:DecodeEncode/sandbox]]. ({{DiffPages|page1=Module:DecodeEncode|page2=Module:DecodeEncode/sandbox}})

''Change:'' apply {{code|1=require('strict')}}, and declade function local explicit. [[User:DePiep|DePiep]] ([[User talk:DePiep|talk]]) 14:34, 21 March 2023 (UTC)

* {{para|answered|'''pause'''}}: needs some extra eyes first. Will invite. -[[User:DePiep|DePiep]] ([[User talk:DePiep|talk]]) 14:36, 21 March 2023 (UTC) 
:[[Wikipedia_talk:Lua#add_require(strict):_code_check_please|Invitation]] is out. -[[User:DePiep|DePiep]] ([[User talk:DePiep|talk]]) 14:49, 21 March 2023 (UTC)
::Upd: Gonnym has made  large improvements, so the sandboxdiff is large. I do not see strict-related changes. [[User:DePiep|DePiep]] ([[User talk:DePiep|talk]]) 21:31, 21 March 2023 (UTC)
:::The changes are good and no globals remain. The two mw.ustring could be string. [[User:Johnuniq|Johnuniq]] ([[User talk:Johnuniq|talk]]) 06:40, 22 March 2023 (UTC)
::::thx. As said, please someone with trust perform ER because me editing/commenting in between does not help. [[User:DePiep|DePiep]] ([[User talk:DePiep|talk]]) 08:18, 22 March 2023 (UTC)
* Set {{para|answered|no}} after two positive critiques. Also, I met no error while developing with this sandbox. -[[User:DePiep|DePiep]] ([[User talk:DePiep|talk]]) 09:00, 22 March 2023 (UTC)
:{{done}}<!-- Template:ETp --> &mdash;&nbsp;Martin <small>([[User:MSGJ|MSGJ]]&nbsp;·&nbsp;[[User talk:MSGJ|talk]])</small> 18:35, 22 March 2023 (UTC)