All X'0D' lost during reading line sequential file using microfocus se
in [Cobol]
|
From: Richard on 29 Jul 2008 03:07 On Jul 29, 3:19 pm, Robert <n...(a)e.mail> wrote: > On Mon, 28 Jul 2008 18:23:18 -0700 (PDT), taoxianf...(a)gmail.com wrote: > >On Jul 29, 2:25 am, "Rick Smith" <ricksm...(a)mfi.net> wrote: > >> <taoxianf...(a)gmail.com> wrote in message > > >>news:a9c0db14-398e-47ef-a705-59e8b9657ba1(a)j22g2000hsf.googlegroups.com.... > > >> [snip] > > >> > BUT now I'm just writing a pure cobol source matching some "special" > >> > text file. > >> > You can do a experiment to see how is a varchar field exported(into > >> > DEL format) - only the characters. > >> > I can't show my data but some records contain no 0D and some have lots > >> > of 0D spreading the whole varchar field. > > >> [snip] > > >> A bit of research (google using: site:ibm.com db2 +aix +hex) > >> shows that a HEX function is available (in DB2 V9) that may be > >> used during EXPORT to convert the characters in the VARCHAR > >> column to hex display. Using this function would seem to simplifiy > >> the problem with having control character values as data. > > >Well, it's really a fantastic method! All data is successfully > >outputted. > > >There's only one more thing: > >I need to import the matched data back to the table, so I'm searching > >the way to convert the HEX displayed varchar column back. Would you > >mind give me a bit more hint? > > Code Page 1252 is hex > > load from filename of del modified by codepage=1252 Do you think that he is running on Windows now ??
From: taoxianfeng on 29 Jul 2008 21:28 On Jul 29, 2:59 pm, "William M. Klein" <wmkl...(a)nospam.netcom.com> wrote: > <taoxianf...(a)gmail.com> wrote in message > > news:71e6da43-8789-4de7-8fc7-54f14fb69dbf(a)34g2000hsh.googlegroups.com... > <snip> > > The DB2 codepage is set to IBM-943 (Japanese) so a SQL2754N "cannot be > converted" error happens when trying to load data with codepage 1252. > Maybe I should change the DB codepage? > > Is the actual mainframe data "NATIONAL" (or DBCS) data (stored in the DB2 table? > If so, then it is CERTAINLY possible that the actual DBCS/national data includes > X"OD" bytes within a double byte (or Unicode) data. > > "Converting" (or handling) mainframe DBCS/National data via Micro Focus on AIX > is a VERY different issue than anything that you have mentioned up to now.. > > If the mainframe data is NOT DBCS or National, can you find out WHY it is > defined as "IBM-943 (Japanese)"? If it does include SOME actual Japanese data, > can you find out if it is ALL "national" - or if it is a mixture of national and > alphanumeric data. > > If the mainframe data includes a combination of EBCDIC and DBCS (or Unicode) > data, then I think you need to be VERY careful of your "conversion" (export) > procedures AND you need to make certain that "conversions" in transfer to AIX > "maintains" valid data AND that you are using the proper language (NLS and > codepage) settings when processing the data with Micro Focus. > > -- > Bill Klein > wmklein <at> ix.netcom.com I just become despaired since it keeps involving more and more questions...
From: Pete Dashwood on 30 Jul 2008 01:37 <taoxianfeng(a)gmail.com> wrote in message news:51a1467d-8e2d-41ab-926b-6732b313957f(a)w7g2000hsa.googlegroups.com... On Jul 29, 2:59 pm, "William M. Klein" <wmkl...(a)nospam.netcom.com> wrote: > <taoxianf...(a)gmail.com> wrote in message > > news:71e6da43-8789-4de7-8fc7-54f14fb69dbf(a)34g2000hsh.googlegroups.com... > <snip> > > The DB2 codepage is set to IBM-943 (Japanese) so a SQL2754N "cannot be > converted" error happens when trying to load data with codepage 1252. > Maybe I should change the DB codepage? > > Is the actual mainframe data "NATIONAL" (or DBCS) data (stored in the DB2 > table? > If so, then it is CERTAINLY possible that the actual DBCS/national data > includes > X"OD" bytes within a double byte (or Unicode) data. > > "Converting" (or handling) mainframe DBCS/National data via Micro Focus on > AIX > is a VERY different issue than anything that you have mentioned up to now. > > If the mainframe data is NOT DBCS or National, can you find out WHY it is > defined as "IBM-943 (Japanese)"? If it does include SOME actual Japanese > data, > can you find out if it is ALL "national" - or if it is a mixture of > national and > alphanumeric data. > > If the mainframe data includes a combination of EBCDIC and DBCS (or > Unicode) > data, then I think you need to be VERY careful of your "conversion" > (export) > procedures AND you need to make certain that "conversions" in transfer to > AIX > "maintains" valid data AND that you are using the proper language (NLS and > codepage) settings when processing the data with Micro Focus. > > -- > Bill Klein > wmklein <at> ix.netcom.com I just become despaired since it keeps involving more and more questions... [Pete] I understand how you feel. I've been watching the thread, but refrained from comment. 1. Don't give up. 2. Think about what you have gained. You have a lot more information than you had when you first posted and you have found out things that you didn't know. Some of the information you received has been misleading, but that's normal on Usenet. People here have been trying hard to help, but the statement of the problem is not accurate. While it may be true that your x'0D' s are being "stripped out", to most people here that is normal behaviour for a Line Sequential file. (that's why it is happening). You didn't tell us the file contained Japanese Language characters which could be represented in a number of ways, and can contain x'0D' as a matter of course. Bill's post above is simply addressing this, and he is trying to help you. (Trust him, he is wise... :-)) Unfortunately, you still haven't been able to resolve your problem, and pressure to do so is mounting. Rick pointed out the possibility of being able to export the data as character format Hex. Very useful. So now, although it all seems very overwhelming, you are really close to a solution. This is not the time to quit or despair... :-) At the moment it seems that as soon as you can reconstruct the original data stream from the Hex, you have solved the problem. How hard can that be? Robert suggested using a code page (unfortunately, he was a bit off the mark, but the idea was good...) Personally, I wouldn't even attempt to change the code page for the DB; that is likely to upset a number of people :-). Think some more about the Hex string. Each byte is represented by 2 hex symbols. If you had to, you could easily write a little COBOL routine that would give you the same byte in binary... Here's an example that is by no means definitive, but which I'm sure you can modify for your particular environment... *> interface 01 two-bytes-in pic xx. 01 one-byte-out pic x. *> reference data 01 hc pic x(16) value '01234567890ABCDEF'. 01 filler redefines hc. 12 hexchars pic x occurs 16 indexed by hc-x1. 01 hv usage comp. 12 x0 pic s9(4) value zero. 12 x1 pic s9(4) value 1. 12 x2 pic s9(4) value 2. 12 x3 pic s9(4) value 3. 12 x4 pic s9(4) value 4. 12 x5 pic s9(4) value 5. 12 x6 pic s9(4) value 6. 12 x7 pic s9(4) value 7. 12 x8 pic s9(4) value 8. 12 x9 pic s9(4) value 9. 12 xA pic s9(4) value 10. 12 xB pic s9(4) value 11. 12 xC pic s9(4) value 12. 12 xD pic s9(4) value 13. 12 xE pic s9(4) value 14. 12 xF pic s9(4) value 15. 01 filler redefines hv. 12 hexvalues pic s9(4) comp occurs 16 indexed by hv-x1. *> work fields 01 current-byte pic x. 01 num-x pic xx. 01 num-b redefines num-x pic s9(4) comp. 01 binary-work-fields usage comp. 12 bin-work pic s9(4). 12 bin-1 pic s9(4). 12 bin-2 pic s9(4). ..... convert-hex-chars section. chc000. move two-bytes-in (1:1) to current-byte perform get-binary move bin-work to bin-1 move two-bytes-in (2:1) to current-byte perform get-binary move bin-work to bin-2 compute num-b = (bin-1 * 16) + bin-2 move num-x (2:1) to one-byte-out . chc999. exit. *>-------------------------- get-binary section. gb000. set hc-x1 to 1 search hexchars at end *> the HEX, isn't... drastic action needed...not shown here when current-byte = hexchars (hc-x1) set hv-x1 to hc-x1 *> you might need to adjust this on MicroFocus move hexvalue (hv-x1) to bin-work end-search . gb999. exit. This is necessarily a little unwieldy because MicroFocus COBOL (as far as I can ascertain) doesn't support PIC 1 for true binary (we really need to address 4 bits here), and that means it is necessary to fudge it in 16 bit fields. If you build a little "machine" (like the above) it isn't too hard to push your HEX string through it and so obtain the original binary representation which could be anything, including National Characters, Katakana, DBCS, whatever. (Or even just standard ASCII) Even if you don't go this way but find another solution, never give up because people are asking questions. Answer the ones you can, ignore the ones you can't or respond with "I don't know"... :-) You have invested a large amount of time and effort in this. You are way too close to a solution to despair now :-) Pete. -- "I used to write COBOL...now I can do anything."
From: taoxianfeng on 30 Jul 2008 03:34 On Jul 30, 2:37 pm, "Pete Dashwood" <dashw...(a)removethis.enternet.co.nz> wrote: > <taoxianf...(a)gmail.com> wrote in message > > news:51a1467d-8e2d-41ab-926b-6732b313957f(a)w7g2000hsa.googlegroups.com... > On Jul 29, 2:59 pm, "William M. Klein" <wmkl...(a)nospam.netcom.com> > wrote: > > > > > > > <taoxianf...(a)gmail.com> wrote in message > > >news:71e6da43-8789-4de7-8fc7-54f14fb69dbf(a)34g2000hsh.googlegroups.com... > > <snip> > > > The DB2 codepage is set to IBM-943 (Japanese) so a SQL2754N "cannot be > > converted" error happens when trying to load data with codepage 1252. > > Maybe I should change the DB codepage? > > > Is the actual mainframe data "NATIONAL" (or DBCS) data (stored in the DB2 > > table? > > If so, then it is CERTAINLY possible that the actual DBCS/national data > > includes > > X"OD" bytes within a double byte (or Unicode) data. > > > "Converting" (or handling) mainframe DBCS/National data via Micro Focus on > > AIX > > is a VERY different issue than anything that you have mentioned up to now. > > > If the mainframe data is NOT DBCS or National, can you find out WHY it is > > defined as "IBM-943 (Japanese)"? If it does include SOME actual Japanese > > data, > > can you find out if it is ALL "national" - or if it is a mixture of > > national and > > alphanumeric data. > > > If the mainframe data includes a combination of EBCDIC and DBCS (or > > Unicode) > > data, then I think you need to be VERY careful of your "conversion" > > (export) > > procedures AND you need to make certain that "conversions" in transfer to > > AIX > > "maintains" valid data AND that you are using the proper language (NLS and > > codepage) settings when processing the data with Micro Focus. > > > -- > > Bill Klein > > wmklein <at> ix.netcom.com > > I just become despaired since it keeps involving more and more > questions... > > [Pete] > > I understand how you feel. I've been watching the thread, but refrained from > comment. > > 1. Don't give up. > 2. Think about what you have gained. > > You have a lot more information than you had when you first posted and you > have found out things that you didn't know. > > Some of the information you received has been misleading, but that's normal > on Usenet. People here have been trying hard to help, but the statement of > the problem is not accurate. While it may be true that your x'0D' s are > being "stripped out", to most people here that is normal behaviour for a > Line Sequential file. (that's why it is happening). You didn't tell us > the file contained Japanese Language characters which could be represented > in a number of ways, and can contain x'0D' as a matter of course. > > Bill's post above is simply addressing this, and he is trying to help you.. > (Trust him, he is wise... :-)) > > Unfortunately, you still haven't been able to resolve your problem, and > pressure to do so is mounting. > > Rick pointed out the possibility of being able to export the data as > character format Hex. Very useful. > > So now, although it all seems very overwhelming, you are really close to a > solution. This is not the time to quit or despair... :-) > > At the moment it seems that as soon as you can reconstruct the original data > stream from the Hex, you have solved the problem. > > How hard can that be? > > Robert suggested using a code page (unfortunately, he was a bit off the > mark, but the idea was good...) > > Personally, I wouldn't even attempt to change the code page for the DB; that > is likely to upset a number of people :-). > > Think some more about the Hex string. Each byte is represented by 2 hex > symbols. If you had to, you could easily write a little COBOL routine that > would give you the same byte in binary... Here's an example that is by no > means definitive, but which I'm sure you can modify for your particular > environment... > > *> interface > 01 two-bytes-in pic xx. > 01 one-byte-out pic x. > > *> reference data > 01 hc pic x(16) value '01234567890ABCDEF'. > 01 filler redefines hc. > 12 hexchars pic x occurs 16 > indexed by hc-x1. > 01 hv usage comp. > 12 x0 pic s9(4) value zero. > 12 x1 pic s9(4) value 1. > 12 x2 pic s9(4) value 2. > 12 x3 pic s9(4) value 3. > 12 x4 pic s9(4) value 4. > 12 x5 pic s9(4) value 5. > 12 x6 pic s9(4) value 6. > 12 x7 pic s9(4) value 7. > 12 x8 pic s9(4) value 8. > 12 x9 pic s9(4) value 9. > 12 xA pic s9(4) value 10. > 12 xB pic s9(4) value 11. > 12 xC pic s9(4) value 12. > 12 xD pic s9(4) value 13. > 12 xE pic s9(4) value 14. > 12 xF pic s9(4) value 15. > 01 filler redefines hv. > 12 hexvalues pic s9(4) comp occurs 16 > indexed by hv-x1. > > *> work fields > > 01 current-byte pic x. > 01 num-x pic xx. > 01 num-b redefines num-x pic s9(4) comp. > 01 binary-work-fields usage comp. > 12 bin-work pic s9(4). > 12 bin-1 pic s9(4). > 12 bin-2 pic s9(4). > > .... > > convert-hex-chars section. > chc000. > move two-bytes-in (1:1) to current-byte > perform get-binary > move bin-work to bin-1 > move two-bytes-in (2:1) to current-byte > perform get-binary > move bin-work to bin-2 > compute num-b = (bin-1 * 16) + bin-2 > move num-x (2:1) to one-byte-out > . > chc999. > exit. > *>-------------------------- > get-binary section. > gb000. > set hc-x1 to 1 > search hexchars > at end > *> the HEX, isn't... drastic action needed...not shown here > when current-byte = hexchars (hc-x1) > set hv-x1 to hc-x1 *> you might need to adjust this on > MicroFocus > move hexvalue (hv-x1) to bin-work > end-search > . > gb999. > exit. > > This is necessarily a little unwieldy because MicroFocus COBOL (as far as I > can ascertain) doesn't support PIC 1 for true binary (we really need to > address 4 bits here), and that means it is necessary to fudge it in 16 bit > fields. > > If you build a little "machine" (like the above) it isn't too hard to push > your HEX string through it and so obtain the original binary representation > which could be anything, including National Characters, Katakana, DBCS, > whatever. (Or even just standard ASCII) > > Even if you don't go this way but find another solution, never give up > because people are asking questions. Answer the ones you can, ignore the > ones you can't or respond with "I don't know"... :-) > > You have invested a large amount of time and effort in this. > > You are way too close to a solution to despair now :-) > > Pete. > -- > "I used to write COBOL...now I can do anything."- Hide quoted text - > > - Show quoted text - You gave an excellent conclusion. I'm really a newbie and I gained a lot from this post. I also think the HEX scalar function is very near to the solution. I'm busy with some other business so replying a little slowly. I will keep trying. Thank you very much.
From: William M. Klein on 30 Jul 2008 06:20
I just want to repeat that if you have a mixture of EBCDIC and National (DBCS or Unicode) data in the DB2 table on the mainframe, that you will really need a LOT of information to be able to "correctly" migrate this to AIX. Question 1: Do you want to convert the EBCDIC data to ASCII? If so, you may still need to find out which EBCDIC code page (there are more than one) is being used on the mainframe. You also won't be able to "automatically" convert the data - as you will NOT want to use the same routine for converting the Japanese data. Question 2: Is the mainframe "Japanese" data in DBCS or Unicode? Do you want it to be in Unicode? Shift-Jis, IBM DBCS, or what format on AIX? IBM mainframe COBOL (and I think - but am not certain DB2) can handle EITHER DBCS or Unicode data. There are differences (some minor and some major) between these. You will need to make certain that you know which format is used on the mainframe AND which format you are supposed to create on AIX. Again, conversions of this data will need to happen "field by field" as you will NOT want to use the same conversion algorithms for this data as the originally EBCDIC data. *** On Windows (but I am not positive it is true on AIX), Micro Focus *does* provide facilities for using mixed EBCDIC and IBM mainframe DBCS data "as if it were native/Windows" data. This *might* be the easies method for doing your conversion/migration work. HOWEVER, it is not recommended for "production" work on AIX. Therefore, you would still want to convert the mainframe-style data to AIX data (i.e. EBCDIC -> ASCII and DBCS -> Unicode) for "production" work. P.S. This is NOT the type of migration that is usually given to a "Newbie" so I certainly can understand your frustration. If this is an 'In-house" migration project, then you should be able to find in-house expertise to help you. If this is something that your organization has "contracted" to do, then I think that someone made a commitment that they hadn't properly "sized" before bidding on. -- Bill Klein wmklein <at> ix.netcom.com <taoxianfeng(a)gmail.com> wrote in message news:7cd6c4c1-3929-418e-b211-9c7e86abd8f3(a)a3g2000prm.googlegroups.com... On Jul 30, 2:37 pm, "Pete Dashwood" <dashw...(a)removethis.enternet.co.nz> wrote: > <taoxianf...(a)gmail.com> wrote in message > > news:51a1467d-8e2d-41ab-926b-6732b313957f(a)w7g2000hsa.googlegroups.com... > On Jul 29, 2:59 pm, "William M. Klein" <wmkl...(a)nospam.netcom.com> > wrote: > > > > > > > <taoxianf...(a)gmail.com> wrote in message > > >news:71e6da43-8789-4de7-8fc7-54f14fb69dbf(a)34g2000hsh.googlegroups.com... > > <snip> > > > The DB2 codepage is set to IBM-943 (Japanese) so a SQL2754N "cannot be > > converted" error happens when trying to load data with codepage 1252. > > Maybe I should change the DB codepage? > > > Is the actual mainframe data "NATIONAL" (or DBCS) data (stored in the DB2 > > table? > > If so, then it is CERTAINLY possible that the actual DBCS/national data > > includes > > X"OD" bytes within a double byte (or Unicode) data. > > > "Converting" (or handling) mainframe DBCS/National data via Micro Focus on > > AIX > > is a VERY different issue than anything that you have mentioned up to now. > > > If the mainframe data is NOT DBCS or National, can you find out WHY it is > > defined as "IBM-943 (Japanese)"? If it does include SOME actual Japanese > > data, > > can you find out if it is ALL "national" - or if it is a mixture of > > national and > > alphanumeric data. > > > If the mainframe data includes a combination of EBCDIC and DBCS (or > > Unicode) > > data, then I think you need to be VERY careful of your "conversion" > > (export) > > procedures AND you need to make certain that "conversions" in transfer to > > AIX > > "maintains" valid data AND that you are using the proper language (NLS and > > codepage) settings when processing the data with Micro Focus. > > > -- > > Bill Klein > > wmklein <at> ix.netcom.com > > I just become despaired since it keeps involving more and more > questions... > > [Pete] > > I understand how you feel. I've been watching the thread, but refrained from > comment. > > 1. Don't give up. > 2. Think about what you have gained. > > You have a lot more information than you had when you first posted and you > have found out things that you didn't know. > > Some of the information you received has been misleading, but that's normal > on Usenet. People here have been trying hard to help, but the statement of > the problem is not accurate. While it may be true that your x'0D' s are > being "stripped out", to most people here that is normal behaviour for a > Line Sequential file. (that's why it is happening). You didn't tell us > the file contained Japanese Language characters which could be represented > in a number of ways, and can contain x'0D' as a matter of course. > > Bill's post above is simply addressing this, and he is trying to help you. > (Trust him, he is wise... :-)) > > Unfortunately, you still haven't been able to resolve your problem, and > pressure to do so is mounting. > > Rick pointed out the possibility of being able to export the data as > character format Hex. Very useful. > > So now, although it all seems very overwhelming, you are really close to a > solution. This is not the time to quit or despair... :-) > > At the moment it seems that as soon as you can reconstruct the original data > stream from the Hex, you have solved the problem. > > How hard can that be? > > Robert suggested using a code page (unfortunately, he was a bit off the > mark, but the idea was good...) > > Personally, I wouldn't even attempt to change the code page for the DB; that > is likely to upset a number of people :-). > > Think some more about the Hex string. Each byte is represented by 2 hex > symbols. If you had to, you could easily write a little COBOL routine that > would give you the same byte in binary... Here's an example that is by no > means definitive, but which I'm sure you can modify for your particular > environment... > > *> interface > 01 two-bytes-in pic xx. > 01 one-byte-out pic x. > > *> reference data > 01 hc pic x(16) value '01234567890ABCDEF'. > 01 filler redefines hc. > 12 hexchars pic x occurs 16 > indexed by hc-x1. > 01 hv usage comp. > 12 x0 pic s9(4) value zero. > 12 x1 pic s9(4) value 1. > 12 x2 pic s9(4) value 2. > 12 x3 pic s9(4) value 3. > 12 x4 pic s9(4) value 4. > 12 x5 pic s9(4) value 5. > 12 x6 pic s9(4) value 6. > 12 x7 pic s9(4) value 7. > 12 x8 pic s9(4) value 8. > 12 x9 pic s9(4) value 9. > 12 xA pic s9(4) value 10. > 12 xB pic s9(4) value 11. > 12 xC pic s9(4) value 12. > 12 xD pic s9(4) value 13. > 12 xE pic s9(4) value 14. > 12 xF pic s9(4) value 15. > 01 filler redefines hv. > 12 hexvalues pic s9(4) comp occurs 16 > indexed by hv-x1. > > *> work fields > > 01 current-byte pic x. > 01 num-x pic xx. > 01 num-b redefines num-x pic s9(4) comp. > 01 binary-work-fields usage comp. > 12 bin-work pic s9(4). > 12 bin-1 pic s9(4). > 12 bin-2 pic s9(4). > > .... > > convert-hex-chars section. > chc000. > move two-bytes-in (1:1) to current-byte > perform get-binary > move bin-work to bin-1 > move two-bytes-in (2:1) to current-byte > perform get-binary > move bin-work to bin-2 > compute num-b = (bin-1 * 16) + bin-2 > move num-x (2:1) to one-byte-out > . > chc999. > exit. > *>-------------------------- > get-binary section. > gb000. > set hc-x1 to 1 > search hexchars > at end > *> the HEX, isn't... drastic action needed...not shown here > when current-byte = hexchars (hc-x1) > set hv-x1 to hc-x1 *> you might need to adjust this on > MicroFocus > move hexvalue (hv-x1) to bin-work > end-search > . > gb999. > exit. > > This is necessarily a little unwieldy because MicroFocus COBOL (as far as I > can ascertain) doesn't support PIC 1 for true binary (we really need to > address 4 bits here), and that means it is necessary to fudge it in 16 bit > fields. > > If you build a little "machine" (like the above) it isn't too hard to push > your HEX string through it and so obtain the original binary representation > which could be anything, including National Characters, Katakana, DBCS, > whatever. (Or even just standard ASCII) > > Even if you don't go this way but find another solution, never give up > because people are asking questions. Answer the ones you can, ignore the > ones you can't or respond with "I don't know"... :-) > > You have invested a large amount of time and effort in this. > > You are way too close to a solution to despair now :-) > > Pete. > -- > "I used to write COBOL...now I can do anything."- Hide quoted text - > > - Show quoted text - You gave an excellent conclusion. I'm really a newbie and I gained a lot from this post. I also think the HEX scalar function is very near to the solution. I'm busy with some other business so replying a little slowly. I will keep trying. Thank you very much. |