Copy and paste into DO command editor - problem with LF

fuzi1968 · July 22, 2015, 3:16pm

I only edit script functions (and all other things) with Ultraedit and store all scripts in Unix format - only LF - not CR/LF.

When I copy & paste the source code into DO command editor, I get e.g.

// OnClick
function OnClick(clickData)
{
  // Get all selected directories
  var enumDirs = new Enumerator(clickData.func.sourcetab.selected_dirs);
  // Call main program
  new FixReleaseAfterRipping(enumDirs);
  // Return ok
  return true;
}

// Library Loader

instead of

// OnClick

function OnClick(clickData)
{
  // Get all selected directories
  var enumDirs = new Enumerator(clickData.func.sourcetab.selected_dirs);

  // Call main program
  new FixReleaseAfterRipping(enumDirs);

  // Return ok
  return true;
}


// Library Loader

Finally 2 LF from my real source will be converted into 1 LF in DO command line editor. 3 LF will result in 2 LF (see last lines of example source).

I think this is a bug (which was always there as I can remember - just using 11.15).

Leo · July 22, 2015, 4:38pm

CRLF is the Windows standard. This isn't really a bug, if it just removes some blank lines when pasting text with i correct (for Windows) line endings.

Opus is a Windows program running on Windows. Why store anything it will use in Unix format?

fuzi1968 · July 22, 2015, 5:11pm

Because I use scripts (e.g. Perl, Oracle) on Unix and Windows and don't save the same source 2 times in 2 different formats. All tools I know are able to handle LF and CR/LF as newline separator (NL).

Ok, I will re-write the question:

The number of empty lines will be reduced by 1 when copying text from Ultraedit (file stored in Unix file format) (Cntrl+C) into Clipboard and then into DO command editor (Ctrl+V):

1 empty line in source file => 0 empty lines in DO
2 empty lines in source file => 1 empty line in DO
3 empty lines in source file => 2 empty lines in DO
...

This Perl regex converts a mix of CR/LF + LF in a string (file) into only LF:

Convert CRLF into LF

$String =~ s/\r?\n/\n/g if defined($String);
or
$String =~ s/\015?\012/\n/g if defined($String);

fuzi1968 · July 22, 2015, 5:15pm

PS:

You want to convert into from a mix of CR/LF + LF into CR/LF .. then you have to use this (Perl regex) of course:

$String =~ s/\r?\n/\r\n/g if defined($String);

NathanPB · July 23, 2015, 3:02am

Is it possible that the original file has a mix of LF and CR/LF line endings (that Ultraedit is then passing through directly to the clipboard)?

I tried two different text editors (neither was UltraEdit) that can properly deal with (and convert between) standard and DOS line ending characters. All line endings in Unix-style pasted into DOpus fine. All line endings in DOS-style pasted into DOpus fine. I used a hex editor to have mixed-line endings in the file, and both editors still pasted into DOpus fine. Using a clipboard viewer that shows the hex dump of the contents (InsideClipboard by Nirsoft) showed that both editors copied the text using DOS-style line endings, regardless of original file encoding.

When I opened the mixed-style file in Notepad (which doesn't show the file correctly) and copied the file, the line endings were send to the clipboard as-is. Only in that case did it paste into DOpus (11.14) the way you describe. Since UltraEdit is a text editor, it probably has more logic to handle slightly weird files than the simplistic editor DOpus has. If the mixed line-endings is the case, either fix the files, or just ignore what it looks like in DOpus since you edit the scripts in UltraEdit anyway.

fuzi1968 · July 23, 2015, 4:52pm

Mixed line endings or not - with the correct implementation (see my regex above) the paste works.

However - there are only Unix line endings (only \n = $0A) in my file. Example part in hex:

000002c0h: 67 28 65 6E 75 6D 44 69 72 73 29 3B 0A 0A 20 20 ; g(enumDirs);..  
000002d0h: 2F 2F 20 52 65 74 75 72 6E 20 6F 6B 0A 20 20 72 ; // Return ok.  r
000002e0h: 65 74 75 72 6E 20 74 72 75 65 3B 0A 7D 0A 0A 0A ; eturn true;.}...
000002f0h: 2F 2F 20 4C 69 62 72 61 72 79 20 4C 6F 61 64 65 ; // Library Loade
00000300h: 72 0A 0A 66 75 6E 63 74 69 6F 6E 20 69 6E 63 6C ; r..function incl

Copy & paste into e.g. a mail (plain text format) also works correctly.

Just tried to find the file where DO stores the text of this script and only found the .dop file

C:\Users\xxx\AppData\Roaming\GPSoftware\Directory Opus\Buttons\MyMusicBar.dop

which includes this:

<instruction>// Clear output</instruction>
<instruction>DOpus.ClearOutput();</instruction>
<instruction>// Include config file</instruction>
<instruction>eval(includeFile(&quot;M:\\Data\\Config\\DoMusic&quot;, &quot;Config.js&quot;));</instruction>
<instruction>// Include main program</instruction>
<instruction>eval(includeFile(DOMUSIC_MAIN_PATH, &quot;FixReleaseAfterRipping.js&quot;));</instruction>
<instruction />
<instruction>// OnClick</instruction>
<instruction>function OnClick(clickData)</instruction>
<instruction>{</instruction>
<instruction>  // Get all selected directories</instruction>
<instruction>  var enumDirs = new Enumerator(clickData.func.sourcetab.selected_dirs);</instruction>
<instruction>  // Call main program</instruction>
<instruction>  new FixReleaseAfterRipping(enumDirs);</instruction>
<instruction>  // Return ok</instruction>
<instruction>  return true;</instruction>
<instruction>}</instruction>
<instruction />
<instruction>// Library Loader</instruction>

When I convert in Ultraedit from Unix line endings to Windows line endings and copy & paste into DO command editor it is correct.

I think Dopus is more then a simplistic editor when it stores the content of soucre code in XML format.

I will try the same on my office PC in a few minutes - where we also have SlickEdit installed.

fuzi1968 · July 23, 2015, 7:09pm

Ok, just tried the same with Slickedit 2014 on my office PC - and it works.

but ...

ok, the file is saved with LF only but I'm not sure how Slickedit handles the file internally in the memory when I load the file. Maybe an auto convert from LF to CR/LF happens and finally the copy & paste is with CR/LF and this works also with Ultraedit!

For "save" I've set LF only in preferences. So during saving it might happen that CR/LF will be converted back to LF.

I was also not able to see the number of characters or bytes in a status line or so - as I can see in Ultraedit. Also not able to search regex for \n or \r\n ... always 0 found. ... btw. I never user Slickedit in my office - it's installed by default.

So - it might work with the editors - but you finally have to know how the data is stored in memory - see above. Which text editors did you use?

The bug only happens when using Ultraedit in Unix line endings and copy & paste into DO command line editor.

I'm not a Windows client programmer and don't know what happens internally when pressing ctrl+C. Is all copied binary or text and will LF converted into CR/LF? Or is the destination program (DO command editor) responsible to convert LF into CR/LF during ctrl+V (paste)? I would think it is the last option - but the DO programmers are the experts and can give us an answer how it works.

fuzi1968 · July 23, 2015, 7:27pm

Ok, I've found it.

The bug must be in DO command editor when getting LF text from clipboard.

I just opened a .sh file on my Unix machine with my SSH terminal program and copied the lines into the DO command editor. One empty line is always lost - see my example above. From 2 empty lines I get 1 in DO. From 3 I get 2, from 4 I get 3 ...

So it does not only happen with/from Ultraedit.

rcoleman1943 · July 24, 2015, 8:14pm

One certainly might wish, even request, that Opus completely supported these files, but isn't it a little strong to keep calling it a bug in Opus when Opus, a Windows program, maybe doesn't completely support files that are not in a Windows standard format?

Maybe it's a bug in Unix to create these files in a format not completely compatible with Windows. No, that's not a completely serious comment, but still the real issue here seems to be that two operating systems simply have different file format standards.

fuzi1968 · July 24, 2015, 9:44pm

DO has no problem to show the Unix LF text files corretly in "Viewer Pane".

It's not a wish and not a bug that DO can not handle LF only into CRLF.

It's only just a small bug during pasting into DO command editor.

And the bug is:

When there are only LF as new line separator in the clipboard one empty line is always missing after pasting into the DO command editor.

Leo · July 25, 2015, 3:18am

Clipboard text data in Windows should have CRLF between lines.

It's not a bug for slightly unusual things to happen with incorrectly formatted inputs.

If anything has a bug, it is the program putting text in that format into the clipboard.

NathanPB · July 26, 2015, 4:16am

On MSDN, I found Standard Clipboard Formats which mentions for clipboard types CF_OEMTEXT (7), CF_TEXT (1), and CF_UNICODETEXT (13) that "Each line ends with a carriage return/linefeed (CR-LF) combination.". So it appears that something that places text on the Windows clipboard that uses a different line-ending scheme is wrong. So if UltraEdit is doing this, that appears to be a bug in UltraEdit.

Did you take a look at the raw hex output after copying from UltraEdit and the terminal program using InsideClipboard by Nirsoft (use Options - Content Display Mode - Hex Dump)? The two text editors I mentioned previously, Notepad, and PuTTY all place text on the clipboard using those three text clipboard types (all of which require the DOS-style line-ending).

(I think the DOS-style line ending is brain-dead, as it has a useless, redundant character, and I think any editor that can't open and save files using the Unix-style line ending is likewise brain-dead. However, since Directory Opus's editor's behavior appears to be from invalid clipboard data, I can't find fault the editor for this reason. (I probably worry too much about whitespace... don't get me started on tabs versus spaces for indentation. ))

fuzi1968 · August 3, 2015, 6:54am

Just back from holiday ...

I previously asked how it works:

but got no detailed reply before I made further tests.

I know that Windows CR/LF is "standard" but it is old and I thought Windows can handle CR/LF and LF internally. I know that Notebad doesn't support LF but I thought it is for compatibility reasons.

Thanks a lot for the superb link! I will forward the info (bug) to Ultraedit.

No, I didn't install the software "InsideClipboard by Nirsoft" (but will do) or "Putty" (I'm using MindTerm).

btw: MindTerm has a setting "copy CR/LF into clipboard" which wasn't enabled.

fuzi1968 · August 3, 2015, 1:27pm

I just installed "inside clipboard" and copied (ctrl+c) from an opened Unix LF file in Ultraedit into clipboard and these hex values are there:

00000000   23 20 43 6C 61 73 73 20 43 6F 6E 73 74 61 6E 74    # Class Constant
00000010   73 0A 75 73 65 20 63 6F 6E 73 74 61 6E 74 20 53    s.use constant S
00000020   54 44 49 4E 5F 4D 41 58 5F 42 55 46 46 45 52 20    TDIN_MAX_BUFFER 
00000030   3D 3E 20 34 30 39 36 3B 0A 0A 0A 23 20 49 6E 73    => 4096;...# Ins
00000040   74 61 6E 63 65 20 4D 65 74 68 6F 64 73 0A 00       tance Methods..

Only 0A (LF) - no CR/LF as expected by MS Standard Clipboard Formats.

I'm still confused that when I paste into DO editor it still makes empty lines from LF only - but always 1 less (1 from 2, 2 from 3, ...). So ... ?

When I paste into Notepad - it does not show any new lines. Full text in one line. Correct (like the definition with CR/LF). DO should do the same as Notepad - only show one line when DO doesn't handle LF only, or isn't it?

btw: I've sent a bug report (same pic and link to this thread) to Ultraedit.

fuzi1968 · August 17, 2015, 6:30am

Hi Leo,

maybe you can re-check this in DO please (not important!).

I did a copy from Ultraedit of an Unix LF file (screen shot left top) into clipboard and there are only LF in the clipboard (see screen shot "insideClipboard" from NirSoft) (okay, a bug of UE so far against the specification - should be CR/LF).

But when I paste it into DO Editor (screen shot right) I don't get the same result as when I paste into Notepad.

Notepad shows one line (screen shot bottom).

DO does not show one line. DO takes account of LF only. Except if there are empty lines. One empty line is always lost. From 1 empty line in DO we see 0. From 2 we see 1. From 3 we see 2, and so on.

However - I've switched internal editing in UE into CR/LF. Then copy + paste works.