Copy files from Serena Network adds hex 00's to files

fuzi1968 · May 5, 2014, 10:06am

Hi,

When I copy (Cobol Source Code) files with DO (10 or 11) from our IBM Host via the Serena Network to my local PC I don't get the correct files.

When I copy them with Windows Explorer (WE) or directly open them with Ultraedit or Slickedit from the Serena Network I get the correct files.

When I check the directory entries from our Serena Network path \CMNP\Produktions-ChangeMan\C07A\Baseline\CSR:

L1NGAU4.CSR ... WE: 243 KB, DO: 248.720 Bytes (242 KB)
LED60.CSR ... WE: 285 KB, DO: 291.840 Bytes (285 KB)
LED60A.CSR ... WE: 29 KB, DO: 29.280 Bytes (28.5 KB)

It looks like the Serena Network reports the column "Size" with the alloction size. There is no other (size) column available via Serena Network.

But the real problem is:

When I copy the files with Windows Explorer to my local PC:

L1NGAU4.CSR ... 96 KB
LED60.CSR ... 152 KB
LED60A.CSR ... 15 KB

When I copy them with Directory Opus to my local PC manually or with this command/button:

COPY FILE "\\CMNP\Produktions-ChangeMan\C07A\Packages\C07A002081\CSR" TO "C:\Data\C07A\2014S06\C07A002081" COPYATTR=no COPYFILETIMES=yes COPYCREATIONTIME=no NONBUFIO=no UPDATEALL WHENEXISTS=replacenewer

(NONBUFIO=no must be set otherwise files > 1 MB (I think depending on the setting) won't be copied without any error or warning).

then I get

L1NGAU4.CSR ... 248.720 Bytes (242 KB)
LED60.CSR ... 291.840 Bytes (285 KB)
LED60A.CSR ... 14.452 Bytes (14.1 KB)

The first two files have the same file size as the directory view above and the third file has the correct file size!

When I open the files directly from Serena Network with Ultraedit or SlickEdit:

L1NGAU4.CSR ... 98.236 Bytes
LED60.CSR ... 155.307 Bytes
LED60A.CSR ... 14.452 Bytes

... like the file size when I copy the files with Windows Explorer to my local PC.

When I open the 2 local files copied with DO in a hex viewer, I can see that they have been filled up with hex 00's until the end - which is wrong. The files copied with Windows Explorer are correct.

It looks like DO copies (bigger) files different against Windows Explorer?

Tested with 11.3, 11.4 but also happend in DO 10 under Windows 7, 64-bit.

Any idea?

Leo · May 5, 2014, 11:15am

Opus just calls ReadFile until the file runs out.

It sounds like Serena Network drives return the wrong data, adding extra zeros on the end which are not meant to be there to pad them out, maybe to a multiple of a sector size or something similar.

If so then the problem is with Serena. Those are incorrect filesystem semantics on Windows.

Given previous issues with Serena Network drives not implementing basic filesystem semantics, this seems a likely explanation for this new problem.

I would report these issues to the Serena Network team as their product appears to have some incorrect behaviour.

fuzi1968 · May 5, 2014, 11:59am

The file size shown in the directory view of DO is correct because Windows Explorer shows the same - maybe Serenta Network wants to show the allocated file size under "Size". So far correct for me.

But why does the copy of files work with Windows Explorer and not with Directory Opus?

When I copy with Windows Explorer from Serena to my local hard disc I have the correct files. With DO only one is correct (maybe only due the smaller file size) although the "Size" in the directory view is different for all 3 files!

I'm not a Windows programer so I have no idea. Does DO use the same functions (API) as WE or not?

The links regarding other (older) problems with Serena Network drivers does not help here.

I can not report a bug to Serena Network unless it works within Windows Explorer. Our company admin contact to Serena Network will not do this in this case.

And if, finally we know what will happen: They will say that Directory Opus does not work correctly (see smaller vs bigger files). It's a never ending bing-bong game.

As long I have no details from deeper inside what could be the reason and how DO works against WE I have no chance to ask for something.

fuzi1968 · May 5, 2014, 12:03pm

See previous comment above too!

Enclosed you can see the screen shot from Windows Explorer and the available columns for the Serena Network View. The properties window of a file does not show the file or allocation size.

Leo · May 5, 2014, 12:28pm

I could also ask: Why does Opus copy correctly using every other filesystem except this one, which we've seen has various other mistakes as well?

Everything you've shown points to the problem being in Serena.

If you really think that Serena will literally only support Windows Explorer, then you should only use Windows Explorer with that drive.

But the same bug in Serena that is causing problems in Opus could also cause problems for other software, so they should want to fix this.

I could write a very simple program which calls ReadFile until the data runs out and prints what it reads, to prove the problem is in Serena, if you want. That would remove Opus from the picture entirely. But there is no point in me writing that for you unless you'd be willing to send the bug report to Serena if my theory turns out to be correct. If the bug is in Serena, which everything points to being the case, then only Serena can fix it.

fuzi1968 · May 5, 2014, 12:58pm

I think as you Leo.

We will get a new OS on our IBM host and finally the new Change Man release and I'm sure a new Serena Network version too.

I think we will wait for the new releases later this year. If the problem still exists or maybe I will ask our "contact admin" later if he would try it when we would have a test program I'll inform you.

I've just asked our admin why we only see the allocated "Size" in the directory view and not the real file size. After copying the files to the local hard discs it looks "wrong".

So I'll wait ... and I will inform you as soon as I have more info.

Thanks a lot for your help!

fuzi1968 · December 4, 2014, 11:05am

Hi,

after a longer time I re-investigated the problem when I copy (Cobol source) files from our IBM host via Serena Network to the local Windows 7 PC.

When files are >= 192 KB, DO (11.8.2 Beta) adds hex 00's to files, Windows Explorer does it not.

The IBM host stores the files as members in a data set which has the record format FB (fixed blocked) and record length 80 bytes. The directory listings (of the members in the data set) in Explorer and DO show the file size: lines * 80, e.g. when a Cobol source has 2471 lines * 80 bytes = 197.680 bytes = 193 KB are displayed (see screen shot for line #2687, LM62A.csr.

Of course the Cobol source has empty lines and shorter lines then 80 characters and when I copy it with Windows Explorer to my local hard disc the correct CRLF endings are stored and finally this file has only a local file size of 100.858 bytes.

But when I copy this file with DO it has 197.680 bytes (same as block size) and fills up file with the difference of 96.822 hex '00' bytes. When I now open this copied file in an editor (e.g. Ultraedit) I see all the hex '00' at the file end. When I directly open the files in Ultraedit from the IBM host/Serena Network path (\CMNP\Produktions-ChangeMan\C07A\Baseline\CSR) - all is ok - correct file.

This only happens when files are >= 192 KB.

Here is screen shot:

The columns

Serena Bytes = Bytes shown in DO directory listing (number of lines of cobol source * 80)
Serena KB = same as above in KB
Serena-Blk-Size = Bytes above / 80 - only integers of course
Explorer Bytes = Bytes shown in DO directory listing after copied the files with Windows Explorer from IBM host to the local PC
DO Bytes = Bytes shown in DO directory listing after copied the files with DO (11.8.2) from IBM host to the local PC
Diff-Explorer-DO = Byte difference

The copy commands in DO are:

CreateFolder NAME="C:\Data\C07A\Baseline"
COPY FILE "\\CMNP\Produktions-ChangeMan\C07A\Baseline\CSR" TO "C:\Data\C07A\Baseline" COPYATTR=no COPYFILETIMES=yes COPYCREATIONTIME=no NONBUFIO=no UPDATEALL WHENEXISTS=replacenewer
COPY FILE "\\CMNP\Produktions-ChangeMan\C07A\Baseline\CBD" TO "C:\Data\C07A\Baseline" COPYATTR=no COPYFILETIMES=yes COPYCREATIONTIME=no NONBUFIO=no UPDATEALL WHENEXISTS=replacenewer

As you can see, files < 192 KB are correctly copied with DO (as Explorer). Files >= 192 KB are NOT correctly copied with DO.

Why does the copy with DO not work correctly? Is there any difference when copying files >= 192 KB?

Thanks a lot!

Leo · December 4, 2014, 2:45pm

You need to report this to Serena, not us.

We have no way to know why their filesystem is adding zeros to data written to it. Only they can diagnose such a problem. All we would see from our side is the proper data being sent to a WriteFile call. The mystery is on the filesystem side of that call, which only the filesystem vendor can diagnose.

fuzi1968 · December 4, 2014, 3:57pm

[quote="leo"]You need to report this to Serena, not us.

We have no way to know why their filesystem is adding zeros to data written to it. Only they can diagnose such a problem. All we would see from our side is the proper data being sent to a WriteFile call. The mystery is on the filesystem side of that call, which only the filesystem vendor can diagnose.[/quote]

Hi Leo,

I will try - but I am sure that our responsible admin for Serena Network will not contact them because under Windows Explorer the problem doesn't occur - and when it works in DO for small files but not for files >= 192 KB - he will say it's a Directory Opus problem.

I am not a Windows programing expert but how does this work with Windows Explorer? Shouldn't Windows Explorer receive the same WriteFile call?