Prev: Live Upgrade - Unable to Determine Boot Device
Next: zfs send/receive... and tape... and stuff...
From: David Combs on 23 Jul 2010 01:02
In article <vsmuf7-uh2.ln1(a)paranoia.mcleod-schmidt.id.au>,
Gary R. Schmidt <grschmidt(a)acm.org> wrote:
>I was just wondering, since I'm about to start rolling new disks into my
>ZFS array, and as a result doubling the size of the zpool and once again
>not being able to back it up onto a single chunk of whatever without
>doing things bit-by-bit, did anyone ever do anything about solving the
>"zfs send to /dev/rmt/0 doesn't work" problem?
>
>Because if they didn't, I may be forced to actually write some real code
>for a change and solve it...
>
>What I am thinking of is basically appropriating the tape-handling code
>from tar/cpio/whatever and using that in a wrapper around zfs send and

Some years ago on this newsgroup there was a long thread on the "tar" version
named STAR. I believe that the concensus was that that its code was
cleaner and bug-freeer than any of the other tars (sun, gnu, ...).

Plus a whole lot more features, needed features, I think.

If you do some of your own coding, why not use STAR as your base?

(I mean, from that long thread, both sun and gnu tar had bugs -- and
now with no more ufsrestore and ufsdump, givcn zfs, those bugs might
now turn out to not be minor AT ALL -- not when doing the most
critical tape-writing at one's site, making backups. So STAR?)

PLEASE, some follow-ups from those who know (and use) STAR --
so we can all learn more about it.

(One very much needed addition to its man-page would be a
whole set of examples, ANNOTATED, using those very, very many
options that program provides.)

THANKS!


David


From: David Combs on 23 Jul 2010 01:36
In article <i0j2bn$gc6i$1(a)tr22n12.aset.psu.edu>,
John D Groenveld <groenvel(a)cse.psu.edu> wrote:
>In article <vsmuf7-uh2.ln1(a)paranoia.mcleod-schmidt.id.au>,
>Gary R. Schmidt <grschmidt(a)acm.org> wrote:
>>So my magic program would chunk up the data from "zfs send" and put
>>headers/footers/checksums and stuff like that on it, write/read the
>>resulting blocks to/from the device, and, when a write/read fails, ask
>>for a new "tape" and carry on from there.
>
>Just beware that a corrupted ZFS stream cannot be received.
>
>I don't know the reliability of your tape streamer, but I suspect
>you're better off using conventional archive tools to backup your
>snapshots to it and only sending your ZFS streams to media that
>has error correction.
>
>John
>groenveld(a)acm.org

Don't modern tape drives (ie for the last 10 or 15 years or so)
already have a lot of error-correction facilities?

EG the Exabyte VXA series, for one?


Or are you talking about something else?

Thanks!

David


From: David Combs on 23 Jul 2010 01:46
In article <aaf6g7-3hs.ln1(a)paranoia.mcleod-schmidt.id.au>,
Gary R. Schmidt <grschmidt(a)acm.org> wrote:
>John D Groenveld wrote:
>> In article <vsmuf7-uh2.ln1(a)paranoia.mcleod-schmidt.id.au>,
>> Gary R. Schmidt <grschmidt(a)acm.org> wrote:
>>> So my magic program would chunk up the data from "zfs send" and put
>>> headers/footers/checksums and stuff like that on it, write/read the
>>> resulting blocks to/from the device, and, when a write/read fails, ask
>>> for a new "tape" and carry on from there.
>>
>> Just beware that a corrupted ZFS stream cannot be received.
>Yes, I was aware of this.
>
>> I don't know the reliability of your tape streamer, but I suspect
>> you're better off using conventional archive tools to backup your
>> snapshots to it and only sending your ZFS streams to media that
>> has error correction.
>This lead me to think about using eSATA/USB disks - create a ZFS file
>system on them and that problem goes away!


You're saying that disks (eSATA/USB, as opposed to scsi?) are a
more robust means of storing critical data?

What about off-site? Wouldn't tapes be easier there?

For LONG-TERM storage (years), I'd heard that these newish
"metal based" tapes simply wouldn't lose data (maybe that's
just the sales pitch).

Especially if you mistakenly drop it -- not much would happen
to a tape (in the plastic box it originally came in), but
a disk, being mechanical, with moving parts, etc, seems
like it might "break"?


Thanks,

David


From: Ian Collins on 23 Jul 2010 01:57
On 07/23/10 05:36 PM, David Combs wrote:
> In article<i0j2bn$gc6i$1(a)tr22n12.aset.psu.edu>,
> John D Groenveld<groenvel(a)cse.psu.edu> wrote:
>> In article<vsmuf7-uh2.ln1(a)paranoia.mcleod-schmidt.id.au>,
>> Gary R. Schmidt<grschmidt(a)acm.org> wrote:
>>> So my magic program would chunk up the data from "zfs send" and put
>>> headers/footers/checksums and stuff like that on it, write/read the
>>> resulting blocks to/from the device, and, when a write/read fails, ask
>>> for a new "tape" and carry on from there.
>>
>> Just beware that a corrupted ZFS stream cannot be received.
>>
>> I don't know the reliability of your tape streamer, but I suspect
>> you're better off using conventional archive tools to backup your
>> snapshots to it and only sending your ZFS streams to media that
>> has error correction.
>>
> Don't modern tape drives (ie for the last 10 or 15 years or so)
> already have a lot of error-correction facilities?
>

All it takes is one bad bit in a zfs stream file and the whole thing is
a goner. Writing zfs stream files to tape is a bad idea, use a
conventional backup tool.

One suggestion that crops up on the zfs lists for those who do want to
write directly to tape is to create a redundant pool out of files and
write those files to tape. Then you have a better chance of recovery if
a file is corrupt.

--
Ian Collins
From: Richard B. Gilbert on 23 Jul 2010 09:21
David Combs wrote:
> In article <vsmuf7-uh2.ln1(a)paranoia.mcleod-schmidt.id.au>,
> Gary R. Schmidt <grschmidt(a)acm.org> wrote:
>> I was just wondering, since I'm about to start rolling new disks into my
>> ZFS array, and as a result doubling the size of the zpool and once again
>> not being able to back it up onto a single chunk of whatever without
>> doing things bit-by-bit, did anyone ever do anything about solving the
>> "zfs send to /dev/rmt/0 doesn't work" problem?
>>
>> Because if they didn't, I may be forced to actually write some real code
>> for a change and solve it...
>>
>> What I am thinking of is basically appropriating the tape-handling code
>>from tar/cpio/whatever and using that in a wrapper around zfs send and
>
> Some years ago on this newsgroup there was a long thread on the "tar" version
> named STAR. I believe that the concensus was that that its code was
> cleaner and bug-freeer than any of the other tars (sun, gnu, ...).
>
> Plus a whole lot more features, needed features, I think.
>
> If you do some of your own coding, why not use STAR as your base?
>
> (I mean, from that long thread, both sun and gnu tar had bugs -- and
> now with no more ufsrestore and ufsdump, givcn zfs, those bugs might
> now turn out to not be minor AT ALL -- not when doing the most
> critical tape-writing at one's site, making backups. So STAR?)
>
> PLEASE, some follow-ups from those who know (and use) STAR --
> so we can all learn more about it.
>
> (One very much needed addition to its man-page would be a
> whole set of examples, ANNOTATED, using those very, very many
> options that program provides.)
>
> THANKS!
>
>
> David
>
>

The author of star used to hang out here but I don't recall that he has
posted here in the last year or two. His name is Georg Schilling
(subject to lapses in my recollection of the correct spelling). You
should be able to find some of his posts in the archives of this news group.

As for the man page, it's traditional to make them terse and cryptic
just like the rest of Unix. After all, the source code IS available. .
.. . ;-)
First  |  Prev  | 
Pages: 1 2
Prev: Live Upgrade - Unable to Determine Boot Device
Next: zfs send/receive... and tape... and stuff...