Prev: how to rebuild single kernel module
Next: Understanding LVM UUIDS
From: Evuraan on 23 Jun 2010 02:00
I've a script which forks and echo's [appends] multiple long lines to
one plain text file. some of the multiple threads also yank lines
from the same plain text file. this file is now approx 31M and has
about 105000 lines ( growing.) i've been doing this for years now,
and have seen nothing go wrong so far. (yay!)


my question is rather theoretical - what can go wrong if multiple
processes are writing to the same file at multiple instances?

i've read elsewhere, that the kernel takes care of this stuff. or is
it that i could just not cause finitely concurrent writes to the file
and have not seen a corrupt line just by luck?

hope this question make sense.

many thanks in advance.


--
To UNSUBSCRIBE, email to debian-user-REQUEST(a)lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster(a)lists.debian.org
Archive: http://lists.debian.org/AANLkTimK7uoDZJI6OevyBLTkmPeRGADIlajLk1yf36Xg(a)mail.gmail.com
From: Ron Johnson on 23 Jun 2010 02:20
On 06/23/2010 12:51 AM, Evuraan wrote:
> I've a script which forks and echo's [appends] multiple long lines to
> one plain text file. some of the multiple threads also yank lines
> from the same plain text file. this file is now approx 31M and has
> about 105000 lines ( growing.) i've been doing this for years now,
> and have seen nothing go wrong so far. (yay!)
>
>
> my question is rather theoretical - what can go wrong if multiple
> processes are writing to the same file at multiple instances?
>
> i've read elsewhere, that the kernel takes care of this stuff. or is
> it that i could just not cause finitely concurrent writes to the file
> and have not seen a corrupt line just by luck?
>

Not only the kernel, but *bash* (or whatever other lesser shell you
use...) is what you'd need to look at to see what it does when it
sees ">>".

> hope this question make sense.
>

It does...

--
Seek truth from facts.


--
To UNSUBSCRIBE, email to debian-user-REQUEST(a)lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster(a)lists.debian.org
Archive: http://lists.debian.org/4C21A755.8030001(a)cox.net
From: Jon Leonard on 23 Jun 2010 02:50
On Wed, Jun 23, 2010 at 01:19:01AM -0500, Ron Johnson wrote:
> On 06/23/2010 12:51 AM, Evuraan wrote:
>> I've a script which forks and echo's [appends] multiple long lines to
>> one plain text file. some of the multiple threads also yank lines
>> from the same plain text file. this file is now approx 31M and has
>> about 105000 lines ( growing.) i've been doing this for years now,
>> and have seen nothing go wrong so far. (yay!)
>>
>>
>> my question is rather theoretical - what can go wrong if multiple
>> processes are writing to the same file at multiple instances?
>>
>> i've read elsewhere, that the kernel takes care of this stuff. or is
>> it that i could just not cause finitely concurrent writes to the file
>> and have not seen a corrupt line just by luck?
>>
>
> Not only the kernel, but *bash* (or whatever other lesser shell you
> use...) is what you'd need to look at to see what it does when it sees
> ">>".

It's using the O_APPEND flag on the writes, which guarantees that each
write(2) system call operates atomically. That is, the data on a given
write will be added to the end of the file, and if two or more processes
are writing to the file at the same time, the results will be consistent
with some ordering. (Data will not be mixed or corrupted between writes.)

It's important to make sure that each line does fit in the same call to
write(), though. I'm not sure what's meant by "long lines": If they're
too big to process all at the same time, the guarantees for a single
call to write() no longer apply. But the shell is almost certainly doing
the right thing. (In C code, I'd check it very carefully.)

Jon Leonard


--
To UNSUBSCRIBE, email to debian-user-REQUEST(a)lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster(a)lists.debian.org
Archive: http://lists.debian.org/20100623063839.GC15502(a)slimy.com
 | 
Pages: 1
Prev: how to rebuild single kernel module
Next: Understanding LVM UUIDS