Need a shell script to run the SAS Jobs
in [SAS]
|
From: Amar Mundankar on 14 May 2010 02:18 Hi All, I want to create a shell script to run the SAS jobs. Mentioned below are the list of jobs that I need to execute every month. There are total 6 steps. In every step I need to execute one or more SAS jobs. Jobs which are within the same step, can be executed parallely. Next step can not be started till the previous step is finished (i.e. Execution of jobs in previous step is finished.) The actual execution process is as follows: EXECUTION PROCESS: Step 1: Run the job FPR_P1_PLNRD.sas (This job takes 1 hour to finish.) Step 2: Once the above job is finished, You can run the following jobs parallelly. (These jobs take 6 to 7 hours to finish.) a) FPR_P1_OAB.sas b) FPR_P1_CDM.sas c) FPR_P1_APA.sas d) FPR_P1_AND.sas e) FPR_P1_ALZ.sas f) FPR_P1_ECS.sas g) FPR_P1_AHY_RD_PCC.sas h) FPR_P1_AHY_RD_CVM.sas Step 3: This step takes around 6 hours to finish. There are total 4 jobs to be run in this step. a) If FPR_P1_AHY_RD_PCC job is finished, you can run the following jobs parallelly. a.1) FPR_P1_AHY_PAY_PCC.sas and a.2) FPR_P1_AHY_TIER_PCC.sas b) If FPR_P1_AHY_RD_CVM.sas job is finished, you can run the following jobs paralley. b.1) FPR_P1_AHY_PAY_CVM.sas and b.2) FPR_P1_AHY_TIER_CVM.sas Step 4: This step takes 2 hours to finish. There is only one job to be run here. a) When both FPR_P1_AHY_RD_PCC.sas (i.e. Step 2 , job g) and FPR_P1_AHY_RD_CVM.sas(i.e. Step 2, job h) jobs are finished, run the following job. a.1) FPR_P1_AHY_DC.sas Stpe 5: This step takes 5 to 6 hours to finish. There are total 2 jobs to be executed. a) If FPR_P1_AHY_PAY_CVM.sas job is finished, run the follwoing job. a.1) FPR_P1_AHY_DOC_CVM.sas b) If FPR_P1_AHY_PAY_PCC.sas job is finished, run the follwoing job. b.1) FPR_P1_AHY_DOC_PCC.sas Step 6: When all the jobs in Step 1 to Step 5 are finished, run the following job. a) FPR_P1_OUT.sas Every month I execute the jobs manually. Hence I have to check whether the currently executing step is finished or not, then only I start the next step. As few steps take 6 to 7 hours to finish, I can not run the whole process in one working day (even if I stay in office for more than 12 hours.) I had created the shell script to run all the jobs one after the other. But it took around 30 hours as I had not run the jobs parallely. The shell script is as follows: # Shell Script to execute the FPR jobs; # Step 1 sas FPR_P1_PLNRD.sas # Step 2 sas FPR_P1_OAB.sas sas FPR_P1_CDM.sas sas FPR_P1_APA.sas sas FPR_P1_AND.sas sas FPR_P1_ALZ.sas sas FPR_P1_ECS.sas sas FPR_P1_AHY_RD_PCC.sas sas FPR_P1_AHY_RD_CVM.sas # Step 3 sas FPR_P1_AHY_PAY_PCC.sas sas FPR_P1_AHY_TIER_PCC.sas sas FPR_P1_AHY_PAY_CVM.sas sas FPR_P1_AHY_TIER_CVM.sas # Step 4 sas FPR_P1_AHY_DC.sas # Step 5 sas FPR_P1_AHY_DOC_CVM.sas sas FPR_P1_AHY_DOC_PCC.sas # Step 6 sas FPR_P1_OUT.sas The problem in above shell script is that all the jobs are executed sequentially. So next sas job is not started till the previous job is finished. I want to write the shell script which will be as per the Execution Process mentioned above. By this I mean, if we look at step 2 then there are total 8 jobs which can be executed parallelly. If I manage to run them parallelly, then I can save some time. I can use the & to run the jobs in step 2 parallelly. e.g. In the shell script, Step 2 can be written as: #Step 2 sas FPR_P1_APA.sas & sas FPR_P1_AND.sas & sas FPR_P1_ALZ.sas & sas FPR_P1_OAB.sas & sas FPR_P1_CDM.sas & sas FPR_P1_ECS.sas & sas FPR_P1_AHY_RD_PCC.sas & sas FPR_P1_AHY_RD_CVM.sas & But for the last command " sas FPR_P1_AHY_RD_CVM.sas & " , if i use the &, then the next job (i.e. sas FPR_P1_AHY_PAY_PCC.sas ) will be started before the previous job is finished. For Step 3, I have to make sure that if any of the last 2 jobs in Step 2 is finished (either CVM or PCC), then only I have to start respective PCC or CVM jobs in Step 3. Similar kind of checks I have to maintain for jobs in Step 4 , 5 and 6. All the checks are mentioned in the Execution Process. The main aim here is to write the shell script which will take into consideration the Execution Process. And will run the jobs accordingly. Any idea?? I am newbie in Shell Script. In Shell script, How can I check whether the particular job is finished or not? and then only I can start the next job. If anybody can write shell script for above process that will be of great help. Any help is appreciated. Thanks in Advance. Regards, Amar Mundankar.
From: RolandRB on 14 May 2010 04:34 On May 14, 8:18 am, Amar Mundankar <amarmundan...(a)gmail.com> wrote: > Hi All, > I want to create a shell script to run the SAS jobs. > > Mentioned below are the list of jobs that I need to execute every > month. > There are total 6 steps. > In every step I need to execute one or more SAS jobs. > Jobs which are within the same step, can be executed parallely. > Next step can not be started till the previous step is finished (i.e. > Execution of jobs in previous step is finished.) > > The actual execution process is as follows: > > EXECUTION PROCESS: > > Step 1: Run the job FPR_P1_PLNRD.sas (This job takes 1 hour to > finish.) > > Step 2: Once the above job is finished, You can run the following jobs > parallelly. (These jobs take 6 to 7 hours to finish.) > a) FPR_P1_OAB.sas > b) FPR_P1_CDM.sas > c) FPR_P1_APA.sas > d) FPR_P1_AND.sas > e) FPR_P1_ALZ.sas > f) FPR_P1_ECS.sas > g) FPR_P1_AHY_RD_PCC.sas > h) FPR_P1_AHY_RD_CVM.sas > > Step 3: This step takes around 6 hours to finish. There are total 4 > jobs to be run in this step. > a) If FPR_P1_AHY_RD_PCC job is finished, you can run the following > jobs parallelly. > a.1) > FPR_P1_AHY_PAY_PCC.sas and > a.2) FPR_P1_AHY_TIER_PCC.sas > > b) If FPR_P1_AHY_RD_CVM.sas job is finished, you can run the > following jobs paralley. > b.1) FPR_P1_AHY_PAY_CVM.sas and > b.2) FPR_P1_AHY_TIER_CVM.sas > > Step 4: This step takes 2 hours to finish. There is only one job to be > run here. > > a) When both FPR_P1_AHY_RD_PCC.sas (i.e. Step 2 , job g) and > FPR_P1_AHY_RD_CVM.sas(i.e. Step 2, job h) jobs are finished, run the > following job. > a.1) FPR_P1_AHY_DC.sas > > Stpe 5: This step takes 5 to 6 hours to finish. There are total 2 jobs > to be executed. > a) If FPR_P1_AHY_PAY_CVM.sas job is finished, run the follwoing job. > a.1) FPR_P1_AHY_DOC_CVM.sas > b) If FPR_P1_AHY_PAY_PCC.sas job is finished, run the follwoing job. > b.1) FPR_P1_AHY_DOC_PCC.sas > Step 6: When all the jobs in Step 1 to Step 5 are finished, run the > following job. > a) FPR_P1_OUT.sas > > Every month I execute the jobs manually. Hence I have to check whether > the currently executing step is finished or not, then only I start > the > next step. As few steps take 6 to 7 hours to finish, I can not run the > whole process in one working day (even if I stay in office for more > than 12 hours.) > I had created the shell script to run all the jobs one after the > other. But it took around 30 hours as I had not run the jobs > parallely. > The shell script is as follows: > > # Shell Script to execute the FPR jobs; > > # Step 1 > sas FPR_P1_PLNRD.sas > > # Step 2 > sas FPR_P1_OAB.sas > sas FPR_P1_CDM.sas > sas FPR_P1_APA.sas > sas FPR_P1_AND.sas > sas FPR_P1_ALZ.sas > sas FPR_P1_ECS.sas > sas FPR_P1_AHY_RD_PCC.sas > sas FPR_P1_AHY_RD_CVM.sas > > # Step 3 > sas FPR_P1_AHY_PAY_PCC.sas > sas FPR_P1_AHY_TIER_PCC.sas > sas FPR_P1_AHY_PAY_CVM.sas > sas FPR_P1_AHY_TIER_CVM.sas > > # Step 4 > sas FPR_P1_AHY_DC.sas > > # Step 5 > sas FPR_P1_AHY_DOC_CVM.sas > sas FPR_P1_AHY_DOC_PCC.sas > > # Step 6 > sas FPR_P1_OUT.sas > > The problem in above shell script is that all the jobs are executed > sequentially. So next sas job is not started till the previous job is > finished. > > I want to write the shell script which will be as per the Execution > Process mentioned above. > By this I mean, if we look at step 2 then there are total 8 jobs which > can be executed parallelly. > If I manage to run them parallelly, then I can save some time. > I can use the & to run the jobs in step 2 parallelly. > > e.g. > In the shell script, Step 2 can be written as: > > #Step 2 > > sas FPR_P1_APA.sas & > sas FPR_P1_AND.sas & > sas FPR_P1_ALZ.sas & > sas FPR_P1_OAB.sas & > sas FPR_P1_CDM.sas & > sas FPR_P1_ECS.sas & > sas FPR_P1_AHY_RD_PCC.sas & > sas FPR_P1_AHY_RD_CVM.sas & > > But for the last command " sas FPR_P1_AHY_RD_CVM.sas & " , if i use > the &, then the next job (i.e. sas FPR_P1_AHY_PAY_PCC.sas ) will be > started > before the previous job is finished. > For Step 3, I have to make sure that if any of the last 2 jobs in Step > 2 is finished (either CVM or PCC), then > only I have to start respective PCC or CVM jobs in Step 3. > > Similar kind of checks I have to maintain for jobs in Step 4 , 5 and > 6. All the checks are mentioned in the Execution Process. > > The main aim here is to write the shell script which will take into > consideration the Execution Process. > And will run the jobs accordingly. > > Any idea?? > I am newbie in Shell Script. > In Shell script, How can I check whether the particular job is > finished or not? and then only I can start the next job. > If anybody can write shell script for above process that will be of > great help. > Any help is appreciated. > > Thanks in Advance. > > Regards, > Amar Mundankar. Firstly, if you are placed in such a job to write shell scripts then you need to be trained to do this and if there is no formal training then you need to be left alone for three months to learn it on your own. Having said the above, you have got the right idea when you use &. What you haven't thought of is breaking the list into several scripts or even having subscripts within those if need be. You can then use & or not with these scripts as need be to give you the order and the parallel running that you need. To give an example, if you have a script named PCC then it could contain the following: # from step 2 sas FPR_P1_AHY_RD_PCC.sas # from step 3 sas FPR_P1_AHY_PAY_PCC.sas & sas FPR_P1_AHY_TIER_PCC.sas & ....and you could have another script named CVM that contains the following: # from step 2 sas FPR_P1_AHY_RD_CVM.sas # from step 3 sas FPR_P1_AHY_PAY_CVM.sas & sas FPR_P1_AHY_TIER_CVM.sas & ....and instead of step g) and h) in step 2 you call these scripts to run in parallel: PCC & CVM & .....and so forth. You should be able to work out the rest yourself.
From: Patrick on 14 May 2010 07:34 As this is a regular task: Can't you use a scheduler? Or may be cron (http://en.wikipedia.org/wiki/Cron).
From: Amar Mundankar on 14 May 2010 08:10 On May 14, 4:34 pm, Patrick <patrick.mat...(a)gmx.ch> wrote: > As this is a regular task: Can't you use a scheduler? Or may be cron > (http://en.wikipedia.org/wiki/Cron). Thanks Roland and Patrick for your responses. Regards, Amar Mundankar.
From: RolandRB on 14 May 2010 12:09 On May 14, 2:10 pm, Amar Mundankar <amarmundan...(a)gmail.com> wrote: > On May 14, 4:34 pm, Patrick <patrick.mat...(a)gmx.ch> wrote: > > > As this is a regular task: Can't you use a scheduler? Or may be cron > > (http://en.wikipedia.org/wiki/Cron). > > Thanks Roland and Patrick for your responses. > > Regards, > Amar Mundankar. Thinking this over then I have changed my mind on how you should approach this. The easiest way for you is to write a separate shell script for each step in the process. Call the shell scripts step1, step2 etc. if you like and then in the controlling script have each step executed one after the other like this: step1 step2 step3 step4 step5 step6 Within each script you can run the jobs in parallel using &
|
Next
|
Last
Pages: 1 2 Prev: Merging 2 datasets by 2 variables Next: "BackTesting" Vintage analysis |