Prev: puts by name
From: Pen Ttt on 13 Aug 2010 21:03
i want to download all usa listed company historial data from yahoo
finance,
at first i download all compamy name list and save it in
/home/pt/usastock/totalname(amex nyse nasdaq,6000 companies),here is my
program to dowload their historial data from yahoo finance with
multi-threads:
require 'rubygems'
require 'open-uri'
threads = []
puts time0=Time.now
i=0
j=0
str=[]
open('/home/pt/usastock/totalname','r'){|item|
while line=item.gets
line=line.chomp
i=i+1
str<<"http://ichart.finance.yahoo.com/table.csv?s="+line
if i==10 or item.eof == true then
begin
for page_to_fetch in str
threads<<Thread.new(page_to_fetch) do |url|
open(url){|urlfile|
datafile="/home/pt/test/"+url.split("=")[1].to_s
open(datafile,'w'){|file|
file.puts urlfile.read
j=j+1
puts "#{url} #{j} is over"
}}
end
end
threads.each {|thr| thr.join}
i=0
str=[]
threads = []
rescue Timeout::Error => e
rescue => e
end
else
end
end
}
puts Time.now-time0
it can run seccessfully,when i change
if i==10 or item.eof == true then (which means i open 10 threads)
into
if i==100 or item.eof == true then (which means i open 100
threads)
there is a problem,some data will not saved in my designated position,
(datafile="/home/pt/axel/"+url.split("=")[1].to_s)
some of them are saved such as "/home/pt/table.csv?s=YHOO"
according to my program,it should be saved as "/home/pt/test/YHOO"
i think that maybe it is a problem of synchronizimg between download
and save
in my program.
would you mind to fix it?
thinks.
--
Posted via http://www.ruby-forum.com/.

 | 
Pages: 1
Prev: puts by name