Prev: Variable length when using Proc Import
Next: RE : Re: copulas and SAS
From: David L Cassell on 2 Oct 2006 01:40
k_nallapeta(a)hotmail.com wrote:
>
>Further to my previous mail, this is the error in the log when I am trying
>to read a 3 GB data file:
>
>----------------------------------------------------------------------------------------------------------------------------------
>588 filename rawfile saszipam 'D:\Kiran\AAPT\Projects\C & VM\Sample MIFs &
>588! DEFs\TEMIAAK20060715115640.zip';
>
>589
>590 data MIF_raw;
>591
>592 /* attrib InArId label="Describes Header Type" format=$1.*/
>593 /* HeReTy label="Describes Header Type" format=$1.*/
>594 /* EnTeNu label="Enquiry Telephone Number Used By Customer"
>format=$11.;*/
>595
>596 infile rawfile(Ebillmth060715) obs=10 missover;
>597 /* Reading Header Information */
>598 /* if _N_ = 1 then do;*/
>599 input @1 InArId 10.
>600 @11 BiIsDt $8.
>601 @19 DoReNu $10.
>602 @29 InReSeNu 9.
>603 @38 SecTyp 2.
>604 @40 LinTyp 1.
>605 @41 HeReTy $1.
>606 @42 EnTeNu $11.
>607 @53 ToLaBi 14.
>608 @67 ToPaRe 14.
>609 @81 TotAdj 14.
>610 @95 ToGsAmAd 14.
>611 @109 ToAmThBi 14.
>612 @123 ToGsToBi 14.
>613 @137 ToAmPa 14.
>614 @151 ToAmOv 14.
>615 @165 PaDuDa $8.
>616 @173 PayExp $60.
>617 @233 CusNam $30.
>618 @263 CuAdLi1 $30.
>619 @293 CuAdLi2 $32.
>620 @325 CuAdLi3 $32.
>621 @357 ComNam $27.
>622 @384 AbnNum $18.
>623 @402 TaInLi $11.
>624 @413 Filler $346.;
>625 /* end;*/
>626 run;
>
>ERROR: Read Access Violation In Task [ DATASTEP )
>Exception occurred at (62A22BE2)
>Task Traceback
>Address Frame (DBGHELP API Version 4.0 rev 5)
>62A22BE2 05C7FC18 saszipam:mcn_main+0x1BE2
>62A22F6F 05C7FCE4 saszipam:mcn_main+0x1F6F
>67990156 05C7FD20 sasxkern:mcn_main+0x3F156
>6723F482 05C7FECC sasdsx:mcn_main+0xE482
>6723EE6B 05C7FEF8 sasdsx:mcn_main+0xDE6B
>6723142A 05C7FF3C sasdsx:mcn_main+0x42A
>673115B0 05C7FF88 sasds:mcn_main+0x5B0
>011B2B02 05C7FFA0 sashost:Main+0xBE72
>011B6C20 05C7FFB4 sashost:Main+0xFF90
>7C80B683 05C7FFEC kernel32:GetModuleFileNameA+0x1B4
>
>NOTE: The SAS System stopped processing this step because of errors.
>WARNING: The data set WORK.MIF_RAW may be incomplete. When this step was
>stopped there were 0
> observations and 26 variables.
>WARNING: Data set WORK.MIF_RAW was not replaced because this step was
>stopped.
>NOTE: DATA statement used (Total process time):
> real time 0.23 seconds
> cpu time 0.07 seconds
>

I think that the problem is your casual mention of the "3 GB" size.

You are doing this on a 32-bit Windows OS. It may not allow you to
have a file over 2 Gigs, and you are (implicitly) creating that full
file and storing it in a temp directory in order to read it in. The
errors may mean that the OS is throwing errors and interrupting
the SAS process, or they may mean that the SASZIPAM file
engine is incapable of handling a file that big.

Try just unzipping the file normally, then reading it. If the OS goes
wonky when you try to store that 3 Gig file, you know that's the
problem. If the OS accepts the file and lets you read it in SAS,
then it is a limitation of the SASZIPAM file.

If you cannot use SASZIPAM but you can read the unzipped file,
you may want to try using a FILENAME PIPE engine instead,
with the pipe being a winzip or pkunzip command. If you cannot
open and work with that full unzipped 3G file, then you need to
try this on a different OS, like linux.

Either way, we needed to know how big your file was when you
first wrote.

HTH,
David
--
David L. Cassell
mathematical statistician
Design Pathways
3115 NW Norwood Pl.
Corvallis OR 97330

_________________________________________________________________
Search?Your way, your world, right now!
http://imagine-windowslive.com/minisites/searchlaunch/?locale=en-us&FORM=WLMTAG
From: "Howard Schreier <hs AT dc-sug DOT org>" on 2 Oct 2006 22:34
On Sun, 1 Oct 2006 22:40:57 -0700, David L Cassell <davidlcassell(a)MSN.COM>
wrote:

>k_nallapeta(a)hotmail.com wrote:
>>
>>Further to my previous mail, this is the error in the log when I am trying
>>to read a 3 GB data file:
>>
>>----------------------------------------------------------------------------------------------------------------------------------
>>588 filename rawfile saszipam 'D:\Kiran\AAPT\Projects\C & VM\Sample MIFs &
>>588! DEFs\TEMIAAK20060715115640.zip';
>>
>>589
>>590 data MIF_raw;
>>591
>>592 /* attrib InArId label="Describes Header Type" format=$1.*/
>>593 /* HeReTy label="Describes Header Type" format=$1.*/
>>594 /* EnTeNu label="Enquiry Telephone Number Used By Customer"
>>format=$11.;*/
>>595
>>596 infile rawfile(Ebillmth060715) obs=10 missover;
>>597 /* Reading Header Information */
>>598 /* if _N_ = 1 then do;*/
>>599 input @1 InArId 10.
>>600 @11 BiIsDt $8.
>>601 @19 DoReNu $10.
>>602 @29 InReSeNu 9.
>>603 @38 SecTyp 2.
>>604 @40 LinTyp 1.
>>605 @41 HeReTy $1.
>>606 @42 EnTeNu $11.
>>607 @53 ToLaBi 14.
>>608 @67 ToPaRe 14.
>>609 @81 TotAdj 14.
>>610 @95 ToGsAmAd 14.
>>611 @109 ToAmThBi 14.
>>612 @123 ToGsToBi 14.
>>613 @137 ToAmPa 14.
>>614 @151 ToAmOv 14.
>>615 @165 PaDuDa $8.
>>616 @173 PayExp $60.
>>617 @233 CusNam $30.
>>618 @263 CuAdLi1 $30.
>>619 @293 CuAdLi2 $32.
>>620 @325 CuAdLi3 $32.
>>621 @357 ComNam $27.
>>622 @384 AbnNum $18.
>>623 @402 TaInLi $11.
>>624 @413 Filler $346.;
>>625 /* end;*/
>>626 run;
>>
>>ERROR: Read Access Violation In Task [ DATASTEP )
>>Exception occurred at (62A22BE2)
>>Task Traceback
>>Address Frame (DBGHELP API Version 4.0 rev 5)
>>62A22BE2 05C7FC18 saszipam:mcn_main+0x1BE2
>>62A22F6F 05C7FCE4 saszipam:mcn_main+0x1F6F
>>67990156 05C7FD20 sasxkern:mcn_main+0x3F156
>>6723F482 05C7FECC sasdsx:mcn_main+0xE482
>>6723EE6B 05C7FEF8 sasdsx:mcn_main+0xDE6B
>>6723142A 05C7FF3C sasdsx:mcn_main+0x42A
>>673115B0 05C7FF88 sasds:mcn_main+0x5B0
>>011B2B02 05C7FFA0 sashost:Main+0xBE72
>>011B6C20 05C7FFB4 sashost:Main+0xFF90
>>7C80B683 05C7FFEC kernel32:GetModuleFileNameA+0x1B4
>>
>>NOTE: The SAS System stopped processing this step because of errors.
>>WARNING: The data set WORK.MIF_RAW may be incomplete. When this step was
>>stopped there were 0
>> observations and 26 variables.
>>WARNING: Data set WORK.MIF_RAW was not replaced because this step was
>>stopped.
>>NOTE: DATA statement used (Total process time):
>> real time 0.23 seconds
>> cpu time 0.07 seconds
>>
>
>I think that the problem is your casual mention of the "3 GB" size.
>
>You are doing this on a 32-bit Windows OS. It may not allow you to
>have a file over 2 Gigs, and you are (implicitly) creating that full
>file and storing it in a temp directory in order to read it in.

I suppose it would have to do that in processing a random access file such
as a native SAS data set, but I would hope that for of a sequential file
(which is the case here) it would be smart enough to sit in the background
and provide one page buffer at a time.

>The errors may mean that the OS is throwing errors and interrupting
>the SAS process, or they may mean that the SASZIPAM file
>engine is incapable of handling a file that big.
>
>Try just unzipping the file normally, then reading it. If the OS goes
>wonky when you try to store that 3 Gig file, you know that's the
>problem. If the OS accepts the file and lets you read it in SAS,
>then it is a limitation of the SASZIPAM file.
>
>If you cannot use SASZIPAM but you can read the unzipped file,
>you may want to try using a FILENAME PIPE engine instead,
>with the pipe being a winzip or pkunzip command. If you cannot
>open and work with that full unzipped 3G file, then you need to
>try this on a different OS, like linux.
>
>Either way, we needed to know how big your file was when you
>first wrote.
>
>HTH,
>David
>--
>David L. Cassell
>mathematical statistician
>Design Pathways
>3115 NW Norwood Pl.
>Corvallis OR 97330
>
>_________________________________________________________________
>Search?Your way, your world, right now!
>http://imagine-windowslive.com/minisites/searchlaunch/?locale=en-us&FORM=WLMTAG
From: Kiran Nallapeta on 3 Oct 2006 00:02
Hi,

I tried reading the file after unzipping it, and it was successfull. It
looks like there is some limitation on SASZIPAM engine for reading huge
files like 3 GB.

All my SAS clients reside on a Windows XP, and SAS Servers reside on Linux.
We do remote submits if we have to excute codes from Base SAS. Normally we
access SAS DI Studio to do ETL but because of limitations in SAS DI, we also
use SAS Base sometimes.

Thanking you and Best Regards,
Kiran


>From: David L Cassell <davidlcassell(a)MSN.COM>
>Reply-To: David L Cassell <davidlcassell(a)MSN.COM>
>To: SAS-L(a)LISTSERV.UGA.EDU
>Subject: Re: Unzip a file
>Date: Sun, 1 Oct 2006 22:40:57 -0700
>
>k_nallapeta(a)hotmail.com wrote:
>>
>>Further to my previous mail, this is the error in the log when I am trying
>>to read a 3 GB data file:
>>
>>----------------------------------------------------------------------------------------------------------------------------------
>>588 filename rawfile saszipam 'D:\Kiran\AAPT\Projects\C & VM\Sample MIFs
>>&
>>588! DEFs\TEMIAAK20060715115640.zip';
>>
>>589
>>590 data MIF_raw;
>>591
>>592 /* attrib InArId label="Describes Header Type" format=$1.*/
>>593 /* HeReTy label="Describes Header Type" format=$1.*/
>>594 /* EnTeNu label="Enquiry Telephone Number Used By Customer"
>>format=$11.;*/
>>595
>>596 infile rawfile(Ebillmth060715) obs=10 missover;
>>597 /* Reading Header Information */
>>598 /* if _N_ = 1 then do;*/
>>599 input @1 InArId 10.
>>600 @11 BiIsDt $8.
>>601 @19 DoReNu $10.
>>602 @29 InReSeNu 9.
>>603 @38 SecTyp 2.
>>604 @40 LinTyp 1.
>>605 @41 HeReTy $1.
>>606 @42 EnTeNu $11.
>>607 @53 ToLaBi 14.
>>608 @67 ToPaRe 14.
>>609 @81 TotAdj 14.
>>610 @95 ToGsAmAd 14.
>>611 @109 ToAmThBi 14.
>>612 @123 ToGsToBi 14.
>>613 @137 ToAmPa 14.
>>614 @151 ToAmOv 14.
>>615 @165 PaDuDa $8.
>>616 @173 PayExp $60.
>>617 @233 CusNam $30.
>>618 @263 CuAdLi1 $30.
>>619 @293 CuAdLi2 $32.
>>620 @325 CuAdLi3 $32.
>>621 @357 ComNam $27.
>>622 @384 AbnNum $18.
>>623 @402 TaInLi $11.
>>624 @413 Filler $346.;
>>625 /* end;*/
>>626 run;
>>
>>ERROR: Read Access Violation In Task [ DATASTEP )
>>Exception occurred at (62A22BE2)
>>Task Traceback
>>Address Frame (DBGHELP API Version 4.0 rev 5)
>>62A22BE2 05C7FC18 saszipam:mcn_main+0x1BE2
>>62A22F6F 05C7FCE4 saszipam:mcn_main+0x1F6F
>>67990156 05C7FD20 sasxkern:mcn_main+0x3F156
>>6723F482 05C7FECC sasdsx:mcn_main+0xE482
>>6723EE6B 05C7FEF8 sasdsx:mcn_main+0xDE6B
>>6723142A 05C7FF3C sasdsx:mcn_main+0x42A
>>673115B0 05C7FF88 sasds:mcn_main+0x5B0
>>011B2B02 05C7FFA0 sashost:Main+0xBE72
>>011B6C20 05C7FFB4 sashost:Main+0xFF90
>>7C80B683 05C7FFEC kernel32:GetModuleFileNameA+0x1B4
>>
>>NOTE: The SAS System stopped processing this step because of errors.
>>WARNING: The data set WORK.MIF_RAW may be incomplete. When this step was
>>stopped there were 0
>> observations and 26 variables.
>>WARNING: Data set WORK.MIF_RAW was not replaced because this step was
>>stopped.
>>NOTE: DATA statement used (Total process time):
>> real time 0.23 seconds
>> cpu time 0.07 seconds
>>
>
>I think that the problem is your casual mention of the "3 GB" size.
>
>You are doing this on a 32-bit Windows OS. It may not allow you to
>have a file over 2 Gigs, and you are (implicitly) creating that full
>file and storing it in a temp directory in order to read it in. The
>errors may mean that the OS is throwing errors and interrupting
>the SAS process, or they may mean that the SASZIPAM file
>engine is incapable of handling a file that big.
>
>Try just unzipping the file normally, then reading it. If the OS goes
>wonky when you try to store that 3 Gig file, you know that's the
>problem. If the OS accepts the file and lets you read it in SAS,
>then it is a limitation of the SASZIPAM file.
>
>If you cannot use SASZIPAM but you can read the unzipped file,
>you may want to try using a FILENAME PIPE engine instead,
>with the pipe being a winzip or pkunzip command. If you cannot
>open and work with that full unzipped 3G file, then you need to
>try this on a different OS, like linux.
>
>Either way, we needed to know how big your file was when you
>first wrote.
>
>HTH,
>David
>--
>David L. Cassell
>mathematical statistician
>Design Pathways
>3115 NW Norwood Pl.
>Corvallis OR 97330
>
>_________________________________________________________________
>Search?Your way, your world, right now!
>http://imagine-windowslive.com/minisites/searchlaunch/?locale=en-us&FORM=WLMTAG

_________________________________________________________________
One and only Ash. Find out all about her. Only on MSN Search
http://server1.msn.co.in/profile/aishwarya.asp
From: David L Cassell on 3 Oct 2006 01:44
Howard Schreier replied:
>
>On Sun, 1 Oct 2006 22:40:57 -0700, David L Cassell <davidlcassell(a)MSN.COM>
>wrote:
>
> >k_nallapeta(a)hotmail.com wrote:
> >>
> >>Further to my previous mail, this is the error in the log when I am
>trying
> >>to read a 3 GB data file:
> >>
> >>----------------------------------------------------------------------------------------------------------------------------------
> >>588 filename rawfile saszipam 'D:\Kiran\AAPT\Projects\C & VM\Sample
>MIFs &
> >>588! DEFs\TEMIAAK20060715115640.zip';
> >>
> >>589
> >>590 data MIF_raw;
> >>591
> >>592 /* attrib InArId label="Describes Header Type" format=$1.*/
> >>593 /* HeReTy label="Describes Header Type" format=$1.*/
> >>594 /* EnTeNu label="Enquiry Telephone Number Used By Customer"
> >>format=$11.;*/
> >>595
> >>596 infile rawfile(Ebillmth060715) obs=10 missover;
> >>597 /* Reading Header Information */
> >>598 /* if _N_ = 1 then do;*/
> >>599 input @1 InArId 10.
> >>600 @11 BiIsDt $8.
> >>601 @19 DoReNu $10.
> >>602 @29 InReSeNu 9.
> >>603 @38 SecTyp 2.
> >>604 @40 LinTyp 1.
> >>605 @41 HeReTy $1.
> >>606 @42 EnTeNu $11.
> >>607 @53 ToLaBi 14.
> >>608 @67 ToPaRe 14.
> >>609 @81 TotAdj 14.
> >>610 @95 ToGsAmAd 14.
> >>611 @109 ToAmThBi 14.
> >>612 @123 ToGsToBi 14.
> >>613 @137 ToAmPa 14.
> >>614 @151 ToAmOv 14.
> >>615 @165 PaDuDa $8.
> >>616 @173 PayExp $60.
> >>617 @233 CusNam $30.
> >>618 @263 CuAdLi1 $30.
> >>619 @293 CuAdLi2 $32.
> >>620 @325 CuAdLi3 $32.
> >>621 @357 ComNam $27.
> >>622 @384 AbnNum $18.
> >>623 @402 TaInLi $11.
> >>624 @413 Filler $346.;
> >>625 /* end;*/
> >>626 run;
> >>
> >>ERROR: Read Access Violation In Task [ DATASTEP )
> >>Exception occurred at (62A22BE2)
> >>Task Traceback
> >>Address Frame (DBGHELP API Version 4.0 rev 5)
> >>62A22BE2 05C7FC18 saszipam:mcn_main+0x1BE2
> >>62A22F6F 05C7FCE4 saszipam:mcn_main+0x1F6F
> >>67990156 05C7FD20 sasxkern:mcn_main+0x3F156
> >>6723F482 05C7FECC sasdsx:mcn_main+0xE482
> >>6723EE6B 05C7FEF8 sasdsx:mcn_main+0xDE6B
> >>6723142A 05C7FF3C sasdsx:mcn_main+0x42A
> >>673115B0 05C7FF88 sasds:mcn_main+0x5B0
> >>011B2B02 05C7FFA0 sashost:Main+0xBE72
> >>011B6C20 05C7FFB4 sashost:Main+0xFF90
> >>7C80B683 05C7FFEC kernel32:GetModuleFileNameA+0x1B4
> >>
> >>NOTE: The SAS System stopped processing this step because of errors.
> >>WARNING: The data set WORK.MIF_RAW may be incomplete. When this step
>was
> >>stopped there were 0
> >> observations and 26 variables.
> >>WARNING: Data set WORK.MIF_RAW was not replaced because this step was
> >>stopped.
> >>NOTE: DATA statement used (Total process time):
> >> real time 0.23 seconds
> >> cpu time 0.07 seconds
> >>
> >
> >I think that the problem is your casual mention of the "3 GB" size.
> >
> >You are doing this on a 32-bit Windows OS. It may not allow you to
> >have a file over 2 Gigs, and you are (implicitly) creating that full
> >file and storing it in a temp directory in order to read it in.

>I suppose it would have to do that in processing a random access file such
>as a native SAS data set, but I would hope that for of a sequential file
>(which is the case here) it would be smart enough to sit in the background
>and provide one page buffer at a time.

> >The errors may mean that the OS is throwing errors and interrupting
> >the SAS process, or they may mean that the SASZIPAM file
> >engine is incapable of handling a file that big.
> >
> >Try just unzipping the file normally, then reading it. If the OS goes
> >wonky when you try to store that 3 Gig file, you know that's the
> >problem. If the OS accepts the file and lets you read it in SAS,
> >then it is a limitation of the SASZIPAM file.
> >
> >If you cannot use SASZIPAM but you can read the unzipped file,
> >you may want to try using a FILENAME PIPE engine instead,
> >with the pipe being a winzip or pkunzip command. If you cannot
> >open and work with that full unzipped 3G file, then you need to
> >try this on a different OS, like linux.
> >
> >Either way, we needed to know how big your file was when you
> >first wrote.
> >
> >HTH,
> >David
> >--
> >David L. Cassell
> >mathematical statistician

Well, on a unix box the pipe mechanism is that smart, most of the
time. But the pipe mechanism in Windows is not. It actually uses
temporary files to hold the resultants, and it is likely to try to expand
the entire file before piping on to the next component. Perhaps
Win32 pipes are getting smarter, but this was true ten years ago.

David
--
David L. Cassell
mathematical statistician
Design Pathways
3115 NW Norwood Pl.
Corvallis OR 97330

_________________________________________________________________
Search?Your way, your world, right now!
http://imagine-windowslive.com/minisites/searchlaunch/?locale=en-us&FORM=WLMTAG
From: David L Cassell on 3 Oct 2006 01:48
k_nallapeta(a)HOTMAIL.COM wrote:
>
>Hi,
>
>I tried reading the file after unzipping it, and it was successfull. It
>looks like there is some limitation on SASZIPAM engine for reading huge
>files like 3 GB.
>
>All my SAS clients reside on a Windows XP, and SAS Servers reside on Linux.
>We do remote submits if we have to excute codes from Base SAS. Normally we
>access SAS DI Studio to do ETL but because of limitations in SAS DI, we
>also
>use SAS Base sometimes.
>
>Thanking you and Best Regards,
>Kiran

Remember that anything running on a 32-bit OS could have an
in-built data file size limitation, just based on available address space
and 32-bit objects used to build the thing.

I would think that you would have a lot more success reading a
zipped file through a pipe on your linux box.

And re-think your DI process. Some of the DI limitations can
be wiggled around. If DI does not seem to do what you need,
go *straight* to SAS Tech Support and ask for help. You are
already paying for that support!

HTH,
David
--
David L. Cassell
mathematical statistician
Design Pathways
3115 NW Norwood Pl.
Corvallis OR 97330

_________________________________________________________________
Find a local pizza place, music store, museum and more?then map the best
route! http://local.live.com
 | 
Pages: 1
Prev: Variable length when using Proc Import
Next: RE : Re: copulas and SAS