Corrupt written data on FAT32 partitions while downloading files

[mst]

Hi, I use ECS1.2R and XP on the same machine. Only ECS has access to the www, so files I need for XP are only downloaded with ECS and stored with the installed fat32.ifs directly at the fat32 drives accessible under both systems.

Since longer time and trying different versions of fat32 (including 0.9.13 latest) I wondered that the downloaded files under XP are corrupt. Selfexpanding exe's are stopping with a CRC error or zip files cannot be expand.

Under ECS I use different versions of Seamonkey (latest 1.1.13)

On my formerly PC I had sometimes the same problems.

I have done some experiments: Downloading the file:  http://startdownload.filefront.com/4286905//6fe471f91cdc109738f8eac44a258057618d7941c6ca8a291bbc33ef2fe926a0a3cdf95c098932f4// (should be project_51.zip ~130MB) direct to a fat32 partition

and a 2nd download to a JFS patition

results in the following:

Download to FAT32 drive: J:\>md5suml project_51.zip 9935917E3288FF884E9B595280BBBCCD project_51.zip filesize 131127388 Bytes

Archive: J:/project_51.zip

inflating: pro51.PK3 error: invalid compressed data to inflate inflating: readme.txt

ERROR

Download to JFS drive: F:\>md5suml project_51.zip 6D6A29E52F1055886A4271E53C3FB968 project_51.zip filesize 131127388 Bytes

Archive: F:/project_51.zip

inflating: pro51.PK3 inflating: readme.txt

OK

So I have written a little file comparison tool. It compares the good and the wrong file and logs all issues. See the attached files. Each line shows the adress, databyte good, databyte bad, total error count and the nummer of continued errors in follow. Please watch the follows carefully!

If I copy such larger files from HPFS or JFS to a FAT32 drive with a filecommander, like FC220, no problem occurs. With my 2Mbit www connection the files are written only slow while downloading them. May be a small buffer allocation problem in the IFS or cachef32.exe?

[mst]

source

[komh]

Hi/2.

What versions of FAT32 were you using ?

And are you using lazy-write cache ? If so, how about without lazy-write cache ? Still corrupted ?

KO Myung-Hun

Replying to mst:

Hi, I use ECS1.2R and XP on the same machine. Only ECS has access to the www, so files I need for XP are only downloaded with ECS and stored with the installed fat32.ifs directly at the fat32 drives accessible under both systems. Since longer time and trying different versions of fat32 (including 0.9.13 latest) I wondered that the downloaded files under XP are corrupt. Selfexpanding exe's are stopping with a CRC error or zip files cannot be expand. Under ECS I use different versions of Seamonkey (latest 1.1.13) On my formerly PC I had sometimes the same problems. I have done some experiments: Downloading the file:  http://startdownload.filefront.com/4286905//6fe471f91cdc109738f8eac44a258057618d7941c6ca8a291bbc33ef2fe926a0a3cdf95c098932f4// (should be project_51.zip ~130MB) direct to a fat32 partition and a 2nd download to a JFS patition results in the following: Download to FAT32 drive: J:\>md5suml project_51.zip 9935917E3288FF884E9B595280BBBCCD project_51.zip filesize 131127388 Bytes Archive: J:/project_51.zip inflating: pro51.PK3 error: invalid compressed data to inflate inflating: readme.txt ERROR Download to JFS drive: F:\>md5suml project_51.zip 6D6A29E52F1055886A4271E53C3FB968 project_51.zip filesize 131127388 Bytes Archive: F:/project_51.zip inflating: pro51.PK3 inflating: readme.txt OK So I have written a little file comparison tool. It compares the good and the wrong file and logs all issues. See the attached files. Each line shows the adress, databyte good, databyte bad, total error count and the nummer of continued errors in follow. Please watch the follows carefully! If I copy such larger files from HPFS or JFS to a FAT32 drive with a filecommander, like FC220, no problem occurs. With my 2Mbit www connection the files are written only slow while downloading them. May be a small buffer allocation problem in the IFS or cachef32.exe?

[mst]

Hi²

I'm actually using this release:  ftp://ftp.netlabs.org/pub/fat32/fat32_0913.wpi

Installed over a older version without changing any parameters 'by hand' in config.sys

I have attached my config.sys

But the defect occurs since ~2 years, also with older versions.

Let me know if you need more info.

Thanks in advance!

[komh]

Hi/2.

You can turn off lazy-write cache by specifying '/L' option to 'cachef32.exe' as the following.

cachef32 /L:off

To turn it on again,

cachef32 /L:on

I want to know the result when turning off lazy-write cache.

KO Myung-Hun

Replying to mst:

Hi² I'm actually using this release:  ftp://ftp.netlabs.org/pub/fat32/fat32_0913.wpi Installed over a older version without changing any parameters 'by hand' in config.sys I have attached my config.sys But the defect occurs since ~2 years, also with older versions. Let me know if you need more info. Thanks in advance!

[mst]

Hi²

I have downloaded the file again. The server was very busy and I needed 1,5h until it was complete. Test with /L:off. File was corrupt, found over 13000 Errors. See attached datvergl2.txt

new download 311208 with cachef32.exe /S /F /L:off J:\>md5suml project_51.zip DA0116023F85A48FBA70E9295FD97246 project_51.zip CRC fail

Archive: J:/project_51.zip

inflating: pro51.PK3 bad CRC a3e8c100 (should be 1c6e2954) inflating: readme.txt

Today I've tested again with /L:on. File was OK, but I had a fast download with 2Mbit/s

new download 020109 with cachef32.exe /S /F /L:on J:\>md5suml project_51.zip 6D6A29E52F1055886A4271E53C3FB968 project_51.zip

Archive: J:/project_51.zip

inflating: pro51.PK3 inflating: readme.txt

OK

I think the problem occurs if only small blocks are written with pauses to a file.

Greetings, Mario

[mst]

=datvergl2.txt

[komh]

Hi/2.

It seems that my explanation is not enough.

You should specify '/L:off' on command line not in config.sys to disable lazy-write cache.

And you should confirm the following message when launching 'cachef32' without any parameters.

LAZY WRITING is NOT active!

Anyway, as you said, small size writing slowly seems to cause the problem.

However, to determine whether this problem occur in file-writing routine or in lazy-write cache routine, is very important.

Thanks.

KO Myung-Hun

Replying to mst:

Hi² I have downloaded the file again. The server was very busy and I needed 1,5h until it was complete. Test with /L:off. File was corrupt, found over 13000 Errors. See attached datvergl2.txt new download 311208 with cachef32.exe /S /F /L:off J:\>md5suml project_51.zip DA0116023F85A48FBA70E9295FD97246 project_51.zip CRC fail Archive: J:/project_51.zip inflating: pro51.PK3 bad CRC a3e8c100 (should be 1c6e2954) inflating: readme.txt Today I've tested again with /L:on. File was OK, but I had a fast download with 2Mbit/s new download 020109 with cachef32.exe /S /F /L:on J:\>md5suml project_51.zip 6D6A29E52F1055886A4271E53C3FB968 project_51.zip Archive: J:/project_51.zip inflating: pro51.PK3 inflating: readme.txt OK I think the problem occurs if only small blocks are written with pauses to a file. Greetings, Mario

[mst]

Hi², I think I catched it!

F:\fat32>cachef32 /L:OFF FAT32 cache helper version 0.9.13. FAT32: WARNING: Lazy writing is not active! CACHEF32 is already running. Current priority is Idle time. LAZY WRITING is NOT active!

Read-Ahead sector count for drive I: is 64. Read-Ahead sector count for drive J: is 64.

CACHE has all space in high memory. CACHE has space for 4096 sectors CACHE contains 4096 sectors There are 0 dirty sectors in cache. The cache hits ratio is 96%. FAT32.IFS has currently 7 GDT segments allocated.

approx 1,5h each download, slow

download 070109 try 1 with commandline: cachef32.exe /L:OFF J:\>md5suml project_51.zip 6D6A29E52F1055886A4271E53C3FB968 project_51.zip CRC = CRC file on JFS OK

download 070109 try 2 with commandline: cachef32.exe /L:OFF J:\>md5suml project_51.zip 6D6A29E52F1055886A4271E53C3FB968 project_51.zip OK

download 070109 try 3 with commandline: cachef32.exe /L:OFF J:\>md5suml project_51.zip 6D6A29E52F1055886A4271E53C3FB968 project_51.zip OK

Date 080109: C:\OS2WARE\FC2_220>cachef32.exe /L:ON FAT32 cache helper version 0.9.13. FAT32: WARNING: Lazy writing is already active or cachesize is 0! CACHEF32 is already running. Current priority is Idle time.

DISKIDLE : 1000 milliseconds. BUFFERIDLE: 500 milliseconds. MAXAGE : 5000 milliseconds.

Read-Ahead sector count for drive I: is 64. Read-Ahead sector count for drive J: is 64.

CACHE has all space in high memory. CACHE has space for 4096 sectors CACHE contains 132 sectors There are 0 dirty sectors in cache. The cache hits ratio is 73%. FAT32.IFS has currently 7 GDT segments allocated.

approx 1,5h each download, slow

J:\>md5suml project_51.zip 0F1C481DBCEB4417B788C477D89F6B1C project_51.zip FAIL!!!!!!!!!!!!!!!!!

After the failure dl. I will switch off the lazy writing. But I miss the line: FAT32: WARNING: Lazy writing is not active''' ???

F:\fat32>cachef32.exe /L:OFF FAT32 cache helper version 0.9.13. CACHEF32 is already running. Current priority is Idle time.

DISKIDLE : 1000 milliseconds. BUFFERIDLE: 500 milliseconds. MAXAGE : 5000 milliseconds.

Read-Ahead sector count for drive I: is 64. Read-Ahead sector count for drive J: is 64.

CACHE has all space in high memory. CACHE has space for 4096 sectors CACHE contains 4096 sectors There are 0 dirty sectors in cache. The cache hits ratio is 96%. FAT32.IFS has currently 7 GDT segments allocated.

I try it again and see the line now. What happens here?

F:\fat32>cachef32.exe /L:OFF FAT32 cache helper version 0.9.13. FAT32: WARNING: Lazy writing is not active! CACHEF32 is already running. Current priority is Idle time. LAZY WRITING is NOT active!

Read-Ahead sector count for drive I: is 64. Read-Ahead sector count for drive J: is 64.

CACHE has all space in high memory. CACHE has space for 4096 sectors CACHE contains 4096 sectors There are 0 dirty sectors in cache. The cache hits ratio is 96%. FAT32.IFS has currently 7 GDT segments allocated.

download time ~2h

J:\>md5suml project_51.zip 6D6A29E52F1055886A4271E53C3FB968 project_51.zip CRC OK

Summary: with L:OFF 4 of 4 downloads are OK with L:ON 1 of 1 download is corrupt cachef32 is still loaded in config.sys: CALL=C:\OS2\CACHEF32.EXE /S /F /L:on

lazy writing is switching on or off by running cachef32 from command line.

I hope this helps. Greetings, Mario.

[komh]

Hi/2.

I greatly appreciate your efforts.

Cache seems to have a problem, but I cannot figure out the problem currently.

Maybe it'll takes very long time. Or other people such as Lars, can solve this problem.

Until then, do not download to FAT32 partition in low speed with cache.

KO Myung-Hun

Replying to mst:

Hi², I think I catched it! F:\fat32>cachef32 /L:OFF FAT32 cache helper version 0.9.13. FAT32: WARNING: Lazy writing is not active! CACHEF32 is already running. Current priority is Idle time. LAZY WRITING is NOT active! Read-Ahead sector count for drive I: is 64. Read-Ahead sector count for drive J: is 64. CACHE has all space in high memory. CACHE has space for 4096 sectors CACHE contains 4096 sectors There are 0 dirty sectors in cache. The cache hits ratio is 96%. FAT32.IFS has currently 7 GDT segments allocated. approx 1,5h each download, slow download 070109 try 1 with commandline: cachef32.exe /L:OFF J:\>md5suml project_51.zip 6D6A29E52F1055886A4271E53C3FB968 project_51.zip CRC = CRC file on JFS OK download 070109 try 2 with commandline: cachef32.exe /L:OFF J:\>md5suml project_51.zip 6D6A29E52F1055886A4271E53C3FB968 project_51.zip OK download 070109 try 3 with commandline: cachef32.exe /L:OFF J:\>md5suml project_51.zip 6D6A29E52F1055886A4271E53C3FB968 project_51.zip OK Date 080109: C:\OS2WARE\FC2_220>cachef32.exe /L:ON FAT32 cache helper version 0.9.13. FAT32: WARNING: Lazy writing is already active or cachesize is 0! CACHEF32 is already running. Current priority is Idle time. DISKIDLE : 1000 milliseconds. BUFFERIDLE: 500 milliseconds. MAXAGE : 5000 milliseconds. Read-Ahead sector count for drive I: is 64. Read-Ahead sector count for drive J: is 64. CACHE has all space in high memory. CACHE has space for 4096 sectors CACHE contains 132 sectors There are 0 dirty sectors in cache. The cache hits ratio is 73%. FAT32.IFS has currently 7 GDT segments allocated. approx 1,5h each download, slow J:\>md5suml project_51.zip 0F1C481DBCEB4417B788C477D89F6B1C project_51.zip FAIL!!!!!!!!!!!!!!!!! After the failure dl. I will switch off the lazy writing. But I miss the line: FAT32: WARNING: Lazy writing is not active''' ??? F:\fat32>cachef32.exe /L:OFF FAT32 cache helper version 0.9.13. CACHEF32 is already running. Current priority is Idle time. DISKIDLE : 1000 milliseconds. BUFFERIDLE: 500 milliseconds. MAXAGE : 5000 milliseconds. Read-Ahead sector count for drive I: is 64. Read-Ahead sector count for drive J: is 64. CACHE has all space in high memory. CACHE has space for 4096 sectors CACHE contains 4096 sectors There are 0 dirty sectors in cache. The cache hits ratio is 96%. FAT32.IFS has currently 7 GDT segments allocated. I try it again and see the line now. What happens here? F:\fat32>cachef32.exe /L:OFF FAT32 cache helper version 0.9.13. FAT32: WARNING: Lazy writing is not active! CACHEF32 is already running. Current priority is Idle time. LAZY WRITING is NOT active! Read-Ahead sector count for drive I: is 64. Read-Ahead sector count for drive J: is 64. CACHE has all space in high memory. CACHE has space for 4096 sectors CACHE contains 4096 sectors There are 0 dirty sectors in cache. The cache hits ratio is 96%. FAT32.IFS has currently 7 GDT segments allocated. download time ~2h J:\>md5suml project_51.zip 6D6A29E52F1055886A4271E53C3FB968 project_51.zip CRC OK Summary: with L:OFF 4 of 4 downloads are OK with L:ON 1 of 1 download is corrupt cachef32 is still loaded in config.sys: CALL=C:\OS2\CACHEF32.EXE /S /F /L:on lazy writing is switching on or off by running cachef32 from command line. I hope this helps. Greetings, Mario.