reading HDF5 dataset subset
in [Matlab]
|
From: Tess on 2 Nov 2009 17:15 Using the low-level HDF5 functions in Matlab, I have generated some HDF5 files with several datasets that are each 512x512x1410. I need to be able to read subsets of these different layers. Looks like hdf5read does not have subsetting capabilities like hdfread, so I am guessing I need to use low-level functions such as H5D.read. How do I translate something like the following high level hdfread call into those HDF5 low-level functions?: data=hdfread('filename','/dataset_name','Index',{[start],[stride],[size]}); e.g. sst=hdfread(file,'/WeeklySST','Index',{[1 1 999],[1 1 1],[512 512 1]}); (Chunk size is set to 512x512x1, but I need to read across those boundaries as well.) Thanks very much, Tess
From: Ashish Uthama on 3 Nov 2009 07:22 On Mon, 02 Nov 2009 17:15:21 -0500, Tess <tess.brandon(a)noaa.gov> wrote: > Using the low-level HDF5 functions in Matlab, I have generated some HDF5 > files with several datasets that are each 512x512x1410. I need to be > able to read subsets of these different layers. Looks like hdf5read > does not have subsetting capabilities like hdfread, so I am guessing I > need to use low-level functions such as H5D.read. How do I translate > something like the following high level hdfread call into those HDF5 > low-level functions?: > > data=hdfread('filename','/dataset_name','Index',{[start],[stride],[size]}); > e.g. > sst=hdfread(file,'/WeeklySST','Index',{[1 1 999],[1 1 1],[512 512 1]}); > > (Chunk size is set to 512x512x1, but I need to read across those > boundaries as well.) > > Thanks very much, > Tess Tess, http://www.hdfgroup.org/ftp/HDF5/examples/examples-by-api/ HDF5 (1.6) http://www.hdfgroup.org/ftp/HDF5/examples/examples-by-api/matlab/HDF5_M_Examples-1.6/h5ex_d_hyper.m
From: Ashish Uthama on 3 Nov 2009 07:24 On Tue, 03 Nov 2009 07:22:26 -0500, Ashish Uthama <first.last(a)mathworks.com> wrote: > On Mon, 02 Nov 2009 17:15:21 -0500, Tess <tess.brandon(a)noaa.gov> wrote: > >> Using the low-level HDF5 functions in Matlab, I have generated some >> HDF5 files with several datasets that are each 512x512x1410. I need to >> be able to read subsets of these different layers. Looks like hdf5read >> does not have subsetting capabilities like hdfread, so I am guessing I >> need to use low-level functions such as H5D.read. How do I translate >> something like the following high level hdfread call into those HDF5 >> low-level functions?: >> >> data=hdfread('filename','/dataset_name','Index',{[start],[stride],[size]}); >> e.g. >> sst=hdfread(file,'/WeeklySST','Index',{[1 1 999],[1 1 1],[512 512 1]}); >> >> (Chunk size is set to 512x512x1, but I need to read across those >> boundaries as well.) >> >> Thanks very much, >> Tess > > Tess, > > http://www.hdfgroup.org/ftp/HDF5/examples/examples-by-api/ > > HDF5 (1.6) > http://www.hdfgroup.org/ftp/HDF5/examples/examples-by-api/matlab/HDF5_M_Examples-1.6/h5ex_d_hyper.m > >
From: Ashish Uthama on 3 Nov 2009 07:27 On Tue, 03 Nov 2009 07:22:26 -0500, Ashish Uthama <first.last(a)mathworks.com> wrote: > On Mon, 02 Nov 2009 17:15:21 -0500, Tess <tess.brandon(a)noaa.gov> wrote: > >> Using the low-level HDF5 functions in Matlab, I have generated some >> HDF5 files with several datasets that are each 512x512x1410. I need to >> be able to read subsets of these different layers. Looks like hdf5read >> does not have subsetting capabilities like hdfread, so I am guessing I >> need to use low-level functions such as H5D.read. How do I translate >> something like the following high level hdfread call into those HDF5 >> low-level functions?: >> >> data=hdfread('filename','/dataset_name','Index',{[start],[stride],[size]}); >> e.g. >> sst=hdfread(file,'/WeeklySST','Index',{[1 1 999],[1 1 1],[512 512 1]}); >> >> (Chunk size is set to 512x512x1, but I need to read across those >> boundaries as well.) >> >> Thanks very much, >> Tess > > Tess, > > http://www.hdfgroup.org/ftp/HDF5/examples/examples-by-api/ > > HDF5 (1.6) > http://www.hdfgroup.org/ftp/HDF5/examples/examples-by-api/matlab/HDF5_M_Examples-1.6/h5ex_d_hyper.m > > Sorry, that went out before I could complete it. The above examples are for MATLAB R2009b and earlier, it would help to have a quick look at http://www.hdfgroup.org/ftp/HDF5/examples/examples-by-api/matlab/HDF5_M_Examples-1.6/README.txt to figure out the correct way to specify the start/stride/count arguments. Please use the 1.8 link for 10a. You could look at the checksum example for specifying chunking. HTH
From: Tess on 3 Nov 2009 14:11
Thanks; I've seen these examples but haven't been able to get them to apply to my problem successfully. Is 'hyperslab' just a fancy word for subset or chunk of the data? Does it matter at all how the data set is chunked? I ran the following code and received the following error: fileID=H5F.open(file); dset=H5D.open(fileID,'/WeeklySST'); % this is a 53x512x512 data set which I % read successfully as a whole using H5D.read space=H5D.get_space(dset); start=[0 0 0]; stride=[50 50 50] % making it symmetrical b/c I'm still figuring out how HDF5 flips % dimensions count=[]; block=[]; % "If count and block are specified as [], the count and block % size default to a single element." H5S.select(space,'H5S_SELECT_SET',start,stride,count,block); ??? Error using ==> hdf5lib2 The HDF5 library encountered an error: "unable to set hyperslab selection" Error in ==> H5S.select_hyperslab at 35 H5ML.hdf5lib2('H5Sselect_hyperslab', spaceID, op, start, stride, count, block); I am running 7.8.0 R2009a. Thanks again, Tess "Ashish Uthama" <first.last(a)mathworks.com> wrote in message <op.u2tjkwbva5ziv5(a)uthamaa.dhcp.mathworks.com>... > On Tue, 03 Nov 2009 07:22:26 -0500, Ashish Uthama > <first.last(a)mathworks.com> wrote: > > > On Mon, 02 Nov 2009 17:15:21 -0500, Tess <tess.brandon(a)noaa.gov> wrote: > > > >> Using the low-level HDF5 functions in Matlab, I have generated some > >> HDF5 files with several datasets that are each 512x512x1410. I need to > >> be able to read subsets of these different layers. Looks like hdf5read > >> does not have subsetting capabilities like hdfread, so I am guessing I > >> need to use low-level functions such as H5D.read. How do I translate > >> something like the following high level hdfread call into those HDF5 > >> low-level functions?: > >> > >> data=hdfread('filename','/dataset_name','Index',{[start],[stride],[size]}); > >> e.g. > >> sst=hdfread(file,'/WeeklySST','Index',{[1 1 999],[1 1 1],[512 512 1]}); > >> > >> (Chunk size is set to 512x512x1, but I need to read across those > >> boundaries as well.) > >> > >> Thanks very much, > >> Tess > > > > Tess, > > > > http://www.hdfgroup.org/ftp/HDF5/examples/examples-by-api/ > > > > HDF5 (1.6) > > http://www.hdfgroup.org/ftp/HDF5/examples/examples-by-api/matlab/HDF5_M_Examples-1.6/h5ex_d_hyper.m > > > > > > Sorry, that went out before I could complete it. > > The above examples are for MATLAB R2009b and earlier, it would help to > have a quick look at > http://www.hdfgroup.org/ftp/HDF5/examples/examples-by-api/matlab/HDF5_M_Examples-1.6/README.txt > to figure out the correct way to specify the start/stride/count arguments. > > Please use the 1.8 link for 10a. > > You could look at the checksum example for specifying chunking. > > HTH |