direct access with Intel fortran
in [Fortran]
|
From: John Paine on 7 Feb 2010 22:22 Hi Glen, Thanks for the comments. I wasn't too sure what to put as the subject, so just opted for the flavor of the task and did not intend it to be interpreted as relating to the access='direct' clause. (Besides, that's also how it is referred to in the Intel documentation, though that's not of a defence). I did include a remark about the possibility of using access='direct', but would prefer not to do that for a number of reasons. Primarily, I'd like the flexibility of rewriting various portions of the binary files I use and frequently the actual position will depend on the data that precedes the location where the data is to be written. I know that I can keep track of this position (and can even use fseek to directly tell me where I am in the file), but wanted to check first that there wasn't some obvious option for the open statement that I was missing that would allow me to rewrite a portion of the file without truncating it. Also, having to have a fixed record length would make it much harder to keep track of where I actually want to write the data and I'd rather revamp my interface to the relevant system calls than rewrite my code to manage extraneous task of counting records. John "glen herrmannsfeldt" <gah(a)ugcs.caltech.edu> wrote in message news:hknlej$4tk$1(a)naig.caltech.edu... > John Paine <johnpaine1(a)optusnet.com.au> wrote: > >> I'm struggling with a problem in trying to perform direct >> access read/write to a file using Intel 11.1. > > (snip) >> 4. Once all of the data blocks have been written, I then want >> to rewrite the header block with the updated statistics and >> data block sizes etc > (snip) > >> open(10,file='test.dat',form='binary') > (snip) > >> write(10)numy,zmin,zmax >> write(10)(numx(j),j=1,numy) > > Your subject says "Direct Access" but your sample code doesn't. > > You need the ACCESS='DIRECT' and RECL= options on the OPEN, > and the REC= on the READ/WRITE statements. RECL specifies > the record length that all records will have. > > If your data doesn't naturally have a fixed length then you > have to do some work to divide it up into fixed length units. > (Possibly wasting the end of some of the records.) > > Note that in the case of statements like: > > write(10)numy,zmin,zmax > write(10)(numx(j),j=1,numy) > > Two records will be used, while it can be written: > > write(10)numy,zmin,zmax,(numx(j),j=1,numy) > > and use only one record. It can then be read with: > > > read(10)numy,zmin,zmax,(numx(j),j=1,numy) > > (But note that these are not direct access I/O statements > without the REC= option. > > -- glen
From: John Paine on 7 Feb 2010 22:46 Hi Louis, Thanks for the response. I did consider this route as it is akin to the separate header/data files, but avoids the problem of separate files. While it is certainly a feasible solution, it's not really one I'd opt for. As per my comments to Glen's response, I was hoping for something that would allow me to simply rewrite data within an existing file using standard fortran and (although unstated in my oriiginal email) with as little rewriting as possible. The task is indeed somewhat similar to the PDF problem as in that case (if I remember correctly) you need to know where the bytes defining the file structure are at the end of the file, and you don't know that position until you've finished writing the contents file. It is a pretty common task to rewrite information in a header block, so I was hoping that Fortran would allow me to do it directly without having to resort to the Windows API or writing the functionality as a C library. Many thanks John "Louis Krupp" <lkrupp_nospam(a)indra.com.invalid> wrote in message news:eZqdnTGRr5zM-PLWnZ2dnUVZ_gednZ2d(a)indra.net... > John Paine wrote: >> Hi all, >> >> I'm struggling with a problem in trying to perform direct access >> read/write to a file using Intel 11.1. >> >> What I want to do is: >> >> 1. Open a data file >> 2. Write a header block to the file defining the data in the file (this >> block is of known size, but unknown content when I open the file) >> 3. Write multiple blocks of binary data of various types sequentially to >> the file and accumulate statistics about the data such as range of >> values, size of data blocks etc >> 4. Once all of the data blocks have been written, I then want to rewrite >> the header block with the updated statistics and data block sizes etc >> 5. Close the file > <snip> > > You may be overlooking the easy way to do this: > > 1. Write your multiple blocks of binary data to a scratch file, > accumulating statistics as you go. > > 2. Rewind the scratch file. > > 3. Open the output data file and write the header with everything you > need. > > 4. Read records from the scratch file and write them to the output file. > > 5. Close the output file. Remove the scratch file. > > Unless your files are really, really big (for some definition of "big"), > this is likely to be fast enough *and* easier to code. Plus, it will let > the header size vary, which might be convenient at some point. > > A couple more thoughts: > > If your header size and/or format change, make sure that new versions of > the file that read the file can read old versions or at least fail > gracefully. Including a version number in the first few bytes of the > header is probably a good way to do this. You'll probably also want to > make sure that old versions of the program know which version(s) of the > file they can read. > > I'm not sure I'd recommend this practice, but as you may know, PDF uses a > cross-reference and a trailer record at the end of the file. Programs > that read PDF files start by reading a bunch of bytes at the end of the > file and scanning for a magic marker string. > > Louis
From: John Paine on 7 Feb 2010 23:25 "James Van Buskirk" <not_valid(a)comcast.net> wrote in message news:hknqce$h1m$1(a)news.eternal-september.org... > "John Paine" <johnpaine1(a)optusnet.com.au> wrote in message > news:4b6f518f$0$32133$afc38c87(a)news.optusnet.com.au... > >> I'm struggling with a problem in trying to perform direct access >> read/write to a file using Intel 11.1. > > C:\gfortran\clf\streamtest>type streamtest.f90 > program streamtest > implicit none > type head > integer i > real x > character(3) c > end type head > type(head) header > integer array(10) > integer i > > array = [(i,i=1,size(array))] > header = head(0,0,'0') > open(10,file='streamtest.dat',access='stream') > write(10) header, array > header = head(2,3.14,'CAT') > write(10,POS=1) header > close(10) > open(10,file='streamtest.dat',access='stream') > header = head(0,0,'0') > array = 0 > read(10), header, array > write(*,*) 'header = ', header > write(*,*) 'array = ', array > end program streamtest > > C:\gfortran\clf\streamtest>gfortran streamtest.f90 -ostreamtest > > C:\gfortran\clf\streamtest>streamtest > header = 2 3.1400001 CAT > array = 1 2 3 4 5 > 6 7 8 9 10 > > -- > write(*,*) transfer((/17.392111325966148d0,6.5794487871554595D-85, & > 6.0134700243160014d-154/),(/'x'/)); end > > Hi James, I haven't had any experience with stream files, so didn't try using that type of access as the Intel documentation didn't sound very promising. However your sample looks like it does the job, so I rewrote the code as below and it does exactly what I want. The one point that troubles me is that the "pos=" keyword uses a number that "indicates a file position in file storage units in a stream file ". This seems to be bytes, but I thought that I saw a comment in the Intel documentation (which I can't locate at the moment), that the "storage unit" is system dependant. Also, I can't find any way to inquire what the unit might be. I'm happy to ignore this for now as it allows me to implement a simple replacement of the API calls with my own equivalents for a stream file, but I'm sure that at some stage the unit size will come back to bite me. Many thanks John implicit none integer*4 i,j integer*4 numy,numx(40) real*4 zmin,zmax real*4 zval(40) c 1: open the file open(10,file='test_stream.dat',access='stream') c 2: initialise and write out the header numy=40 zmin=1e32 zmax=-1e32 do j=1,numy numx(j)=0 end do write(10)numy,zmin,zmax,(numx(j),j=1,numy) c 3: loop over the data do j=1,numy c do stuff to create numx(j) data values in array zval c NB numx(j) is determined as part of the calculation numx(j)=j do i=1,numx(j) zval(i)=i zmin=min(zmin,zval(i)) zmax=max(zmax,zval(i)) end do c write out the data write(10)(zval(i),i=1,numx(j)) end do c 4: rewrite the changed avlues in the header write(10,pos=5)zmin,zmax,(numx(j),j=1,numy) c 5: close the file close(10) c reopen the file and read the data open(10,file='test_stream',access='stream') c this step works ok and the header is read correctly read(10)numy,zmin,zmax,(numx(j),j=1,numy) c this loop correctly reads the data from the file do j=1,numy read(10)(zval(i),i=1,numx(j)) end do close(10) end
From: John Paine on 7 Feb 2010 23:33 "Richard Maine" <nospam(a)see.signature> wrote in message news:1jdk4w1.162yhg4tspypsN%nospam(a)see.signature... > John Paine <johnpaine1(a)optusnet.com.au> wrote: > >> What I want to do is: > .... >> 4. Once all of the data blocks have been written, I then want to rewrite >> the >> header block with the updated statistics and data block sizes etc > ... >> In CVF, I used to do this using the dfwin interface > .. >> Since I am now migrating my code to the Intel >> compiler, I'm cleaning up my code base and would like to eliminate the >> Windows routines and use standard Fortran IO. > [example using form='binary'] > > Well form='binary' is not standard Fortran, so if that is the objective, > this would not achieve it even if it did act like you wanted. > I knew I was vulnerable on that front and was hoping that nobody would notice. I know it's not standard, but I'm assuming that it'd be a brave vendor who left it out in the forseeable future. >> This all works fine, except that step 4 truncates the file after the >> write >> so all of the data records are lost.... >> The behavior of the 'binary' write truncating the file is documentated by >> Intel. They also include a mention of "direct access" which suggests that >> direct access be used. > > Yes, you could certainly do it with direct access, but there are lots of > complications - more than you mentioned. For a start, > >> step 1 with access='direct',recl=1 specified. > > The use of recl=1 with direct access is a hack recognized by some > compilers, but it is not standard. Direct access is standard, but the > common special-case interpretation of recl=1 is not. There are other > complications as well. > I agree, and that was one of the considerations for not going down that route. Though if it had worked (and I did try it), I wouldn't have been posting. >> Thanks in advance for any suggestions. > > I'd recommend against going with direct access. It certainly can be > done; I've done that kind of thing in the past. But the many gotchas of > direct access are a large part of why I was a big pusher for stream > access in f2003. > > I recommend using stream access. James gave some presumably fine example > code (I didn't check in detail, but I suspect it is fine). I just > thought I'd supply some English to supplement his Fortran. :-) > > The form='binary' and access='direct',recl=1 are both nonstandard > variants of stream access. They date from before stream was > standardized. Now that stream is standardized, I recommend using it > instead of those nonstandard variants. As an additional "side" benefit, > the standard requires that it act like you want (no truncation), at > least as long as you stick to unformatted stream. (Formatted is a > different story). > > The approach that Louis mentioned (using a temporary intermediate file) > can also work well; that's your choice to make. I will be going down the stream route and James' code was more than adequate to put me on the right path. As an added benefit, it now seems that I will be using standard fortran to do what I want. Very pleasing! > > -- > Richard Maine | Good judgment comes from experience; > email: last name at domain . net | experience comes from bad judgment. > domain: summertriangle | -- Mark Twain
From: Richard Maine on 8 Feb 2010 01:50
John Paine <johnpaine1(a)optusnet.com.au> wrote: > "Richard Maine" <nospam(a)see.signature> wrote in message > news:1jdk4w1.162yhg4tspypsN%nospam(a)see.signature... > > Well form='binary' is not standard Fortran, so if that is the objective, > > this would not achieve it even if it did act like you wanted. > > > I knew I was vulnerable on that front and was hoping that nobody would > notice. I know it's not standard, but I'm assuming that it'd be a brave > vendor who left it out in the forseeable future. Lots of vendors already don't have it. Pretty much all of them have some kind of stream-like functionality, but the details do vary already. If a large majority of vendors already did it the same way, that way probably would have been adopted as the standard instead of introducing stream. But they don't, so it wasn't. Now if you are talking about the odds of Intel in specific dropping it, I'd agree those odds would be low. But the odds of some other random vendor happening to have it are much more questionable. (I see you are happy with the suggestion to use stream, so these points are perhaps a bit academic). -- Richard Maine | Good judgment comes from experience; email: last name at domain . net | experience comes from bad judgment. domain: summertriangle | -- Mark Twain |