problems reading binary file
in [Java Programming]
|
From: Mikee on 9 Jun 2010 03:17 Hi I having trouble reading a binary file. The problem seems to occur with reading some float values so I've made a small test file (from SQL server using native file output) that contains just 5 floats. -13.73482 , 20.689 , -99.99999 , 20.157 ,1 9.454 I read the file in and write out the values using DataInputStream in = new DataInputStream(new BufferedInputStream(new FileInputStream(filename))); System.out.print(ByteSwapper.swap(in.readFloat())+" "); System.out.print(ByteSwapper.swap(in.readFloat())+" "); System.out.print(ByteSwapper.swap(in.readFloat())+" "); System.out.print(ByteSwapper.swap(in.readFloat())+" "); System.out.println(ByteSwapper.swap(in.readFloat())+" "); The output is -13.734822 20.689 6.9055E-41 20.157 19.454 ie -99.99999 is now 6.9055E-41. I'm trying to read in a much larger SQL outgest of mixed data types integers, longs, doubles, floats and bytes and the only problem I've spotted is floats of -99.9999 (which is a deafult value used for some columns) It was necessary to byte swap and ByteSwapper.swap(float) is something I find on the web an dis below. I tries other byte swap code with the same result. Any ideas what I'm doing wrong with the 3rd value and presumably in general. Python code of binaryFile = file('test.bin', 'rb') while True: rowBinary = binaryFile.read(20) if not rowBinary: break print struct.unpack('<5f', rowBinary) binaryFile.close() correctly gives -13.734822273254395, 20.688999176025391, -99.999992370605469, 20.156999588012695, 19.454000473022461 java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_12-b04 Thanks Mike public static float swap (float value) { int intValue = Float.floatToIntBits (value); intValue = swap (intValue); return Float.intBitsToFloat (intValue); } and public static int swap (int value) { int b1 = (value >> 0) & 0xff; int b2 = (value >> 8) & 0xff; int b3 = (value >> 16) & 0xff; int b4 = (value >> 24) & 0xff; return b1 << 24 | b2 << 16 | b3 << 8 | b4 << 0; }
From: Thomas Pornin on 9 Jun 2010 06:14 According to Mikee <mikee.read(a)googlemail.com>: > DataInputStream in = new DataInputStream(new > BufferedInputStream(new FileInputStream(filename))); > System.out.print(ByteSwapper.swap(in.readFloat())+" "); > System.out.print(ByteSwapper.swap(in.readFloat())+" "); > System.out.print(ByteSwapper.swap(in.readFloat())+" "); > System.out.print(ByteSwapper.swap(in.readFloat())+" "); > System.out.println(ByteSwapper.swap(in.readFloat())+" "); > [...] > public static float swap (float value) > { > int intValue = Float.floatToIntBits (value); > intValue = swap (intValue); > return Float.intBitsToFloat (intValue); > } This is conceptually flawed. What you do, here, is the following: 1. read four bytes 2. interpret them as a float (in.readFloat()) 3. convert them back to four bytes, as an "int" value (Float.floatToIntBits()) 4. byte-swap the bytes 5. finally reinterpret the bytes as a float (Float.intBitsToFloat()) Steps 2 and 3 basically cancel each other: step 3 undoes what step 2 did. The trouble is that this roundtrip (steps 2 and 3) may lose some information, because some bit patterns encode a "NaN" (a special value for floats) and Float.floatToIntBits() normalizes NaN values. This is probably what happens to you: when read "byte-swapped", your -99.99999 becomes a pattern for a NaN, which gets normalized to another NaN, thus changing the bit pattern. You _might_ correct your code by using Float.float.toRawIntBits() instead of Float.floatToIntBits(), but even if it works, it is non-robust (it assumes that DataInputStream also does things that way). A more correct way of doing things is to read the bytes as an "int" directly, which basically removes steps 2 and 3 altogether. This would look like this: System.out.print(Float.intBitsToFloat(swap(in.readInt())) + " "); --Thomas Pornin
From: Paul Cager on 9 Jun 2010 06:53 On Jun 9, 11:14 am, Thomas Pornin <por...(a)bolet.org> wrote: > According to Mikee <mikee.r...(a)googlemail.com>: > > > DataInputStream in = new DataInputStream(new > > BufferedInputStream(new FileInputStream(filename))); > > System.out.print(ByteSwapper.swap(in.readFloat())+" "); [...] > You _might_ correct your code by using Float.float.toRawIntBits() > instead of Float.floatToIntBits(), but even if it works, it is > non-robust (it assumes that DataInputStream also does things that way). > A more correct way of doing things is to read the bytes as an "int" > directly, which basically removes steps 2 and 3 altogether. This would > look like this: > > System.out.print(Float.intBitsToFloat(swap(in.readInt())) + " "); I haven't checked the API (http://java.sun.com/javase/6/docs/api/ index.html?java/nio/package-summary.html) but I seem to remember that NIO allows you to read floats etc in the byte order of your choice. That might avoid the need to do all the bit twiddling yourself.
From: Mikee on 9 Jun 2010 09:03 Hi Thomas Thanks for poitning out where I was going wrong. > System.out.print(Float.intBitsToFloat(swap(in.readInt())) + " "); Yes that worked, Thanks again Mike
From: Mikee on 9 Jun 2010 09:05
Thanks Paul Currently part of the environment I'm working in is constrained to 1.5 but I'lll check 1.6 out for future ref. > I haven't checked the API (http://java.sun.com/javase/6/docs/api/ > index.html?java/nio/package-summary.html) but I seem to remember that > NIO allows you to read floats etc in the byte order of your choice. > That might avoid the need to do all the bit twiddling yourself. Cheers Mike |