Command line (CLI) environment setup

To set up the command line output (conman, composer) there are 3 variables that can be adjusted to obtain the optimum output format.

MAESTROLINES: Specifies the number of lines per screen. At the end of each screen page, conman prompts to continue. If MAESTROLINES (or LINES) is set to zero or a negative number, conman does not pause at the end of a page. 

MAESTROCOLUMNS: Specifies the number of characters per line. The following options are available: Less than 120 (only some columns will be displayed). Equal to or more than 120 (all columns will be displayed). The maximum value is 1024. If either MAESTROLINES or MAESTROCOLUMNS is not set, the standard shell variables, LINES and COLUMNS, are used. 

MAESTRO_OUTPUT_STYLE: Specifies how object names are displayed. If set to LONG, full names are displayed. If not set, or set to any value other than LONG, long names are truncated to eight characters followed by a plus sign (+). 

Default unix output: 

       
bash-3.00$ ./<TWS_home>/bin/conman sj @#final
Tivoli Workload Scheduler (UNIX)/CONMAN 8.6.0.00 (20110719)
Licensed Materials - Property of IBM*
5698-WSH
(C) Copyright IBM Corp. 1998, 2011 All rights reserved.
* Trademark of International Business Machines
Installed for user "maestro".
Locale LANG set to the following: "en"
Scheduled for (Exp) 04/04/12 (#89) on TW1T.  Batchman LIVES.  Limit: 20, Fence: 0, Audit Level: 1
%sj @#final
                                               (Est)  (Est)
CPU      Schedule SchedTime  Job      State Pr Start  Elapse RetCode Deps

TW1T    #FINAL    0003 04/04 ******** SUCC  10 00:03  00:01          [04/03/12]
                             STARTAP+ SUCC  10 00:03  00:01        0 #J18294
                             MAKEPLAN SUCC  10 00:03  00:01        0 #J18468
                             SWITCHP+ SUCC  10 00:03  00:01        0 #J18583
                             CREATEP+ SUCC  10 00:04  00:01        0 #J18990
                             UPDATES+ SUCC  10 00:04  00:01        0 #J18996

TW1T    #FINAL    0003 04/05 ******** HOLD  10(04/05)(00:05)         [Carry]
                             STARTAP+ HOLD  10(04/05)(00:01)
                             MAKEPLAN HOLD  10(04/05)(00:01)         STARTAP+
                             SWITCHP+ HOLD  10(04/05)(00:01)         MAKEPLAN
                             CREATEP+ HOLD  10(04/05)(00:01)         SWITCHP+
                             UPDATES+ HOLD  10(04/05)(00:01)         SWITCHP+

And the output with variables set as: 
MAESTRO_OUTPUT_STYLE=LONG
MAESTROLINES=0
MAESTROCOLUMS=512

       
bash-3.00$ ./<TWS_home>/bin/conman sj @#final
Tivoli Workload Scheduler (UNIX)/CONMAN 8.6.0.00 (20110719)
Licensed Materials - Property of IBM*
5698-WSH
(C) Copyright IBM Corp. 1998, 2011 All rights reserved.
* Trademark of International Business Machines
Installed for user "maestro".
Locale LANG set to the following: "en"
Scheduled for (Exp) 04/04/12 (#89) on TW1T.  Batchman LIVES.  Limit: 20, Fence: 0, Audit Level: 1
%sj @#@final
                                                                                               (Est)  (Est)
Workstation      Job Stream       SchedTime  Job                                      State Pr Start  Elapse ReturnCode Dependencies

TW1T            #FINAL            0003 04/04 **************************************** SUCC  10 00:03  00:01             [04/03/12]
                                             STARTAPPSERVER                           SUCC  10 00:03  00:01           0 #J18294
                                             MAKEPLAN                                 SUCC  10 00:03  00:01           0 #J18468
                                             SWITCHPLAN                               SUCC  10 00:03  00:01           0 #J18583
                                             CREATEPOSTREPORTS                        SUCC  10 00:04  00:01           0 #J18990
                                             UPDATESTATS                              SUCC  10 00:04  00:01           0 #J18996

TW1T            #FINAL            0003 04/05 **************************************** HOLD  10(04/05)(00:05)            [Carry]
                                             STARTAPPSERVER                           HOLD  10(04/05)(00:01)
                                             MAKEPLAN                                 HOLD  10(04/05)(00:01)            STARTAPPSERVER
                                             SWITCHPLAN                               HOLD  10(04/05)(00:01)            MAKEPLAN
                                             CREATEPOSTREPORTS                        HOLD  10(04/05)(00:01)            SWITCHPLAN
                                             UPDATESTATS                              HOLD  10(04/05)(00:01)            SWITCHPLAN

To have the variables set by default, add the following lines to your user profile file (.profile, .bash_profile etc): 

       
<TWS_home>/tws_env.sh
export MAESTRO_OUTPUT_STYLE=LONG
export MAESTROLINES=0
export MAESTROCOLUMS=512

FTA Stop & Start commands & script

To Start & Stop a TWS FTA use the below commands: 

1. Start a TWS FTA:

       
./<TWS_home>/StartUp
./<TWS_home>/bin/conman "start & link @;noask"

2. Stop a TWS FTA:

       
./<TWS_home>/bin/conman "shutdown;wait"

And a small script to start / stop TWS FTA:

       
#!/bin/sh
#
# Start/Stop the Tivoli Workload Scheduler FTA
#

case "$1" in

   start)
        /<TWS_home>/StartUp
        /<TWS_home>/bin/conman "start & link @;noask"
        ;;
   stop)
        /<TWS_home>/bin/conman "shutdown;wait"
        ;;
   *)
      echo "Unknown flag passed to $0."
      echo "Usage: $0 {start, stop}"

esac
exit 0

Note: The above script is the simplest possible, I like simple things, even we have a error as it will try to start / stop all the agents but it does not have rights, which is normal.

Jobs migration between agents

The easiest way to migrate jobs / job streams between the agents without any impact and complete invisible for end users is to use composer commands as below:

1. Create a dump containing all the jobs & the job streams from the source agent (S_FTA):

       
./<TWS_home>/bin/composer create /tmp/S_FTA_all_jobstreams.txt sched=S_FTA#@
./<TWS_home>/bin/composer create /tmp/S_FTA_all_jobs.txt jobs=S_FTA#@

2. Rename the source (S_FTA) agent with destination agent (D_FTA) and save the files using a different name (D_FTA_all_jobstreams.txt & D_FTA_all_jobs.txt) so you can restore it in case is needed (you can use any text converter you want).

3. Remove the dependencies using temporary files (if the dependencies exist when importing the job streams we will get an error message and the job stream is not imported, and we need to import them again and again until all the dependencies are met, or it will not work at all if we have a close cycle):

       
cat /tmp/D_FTA_all_jobstreams.txt | grep -v "FOLLOWS" > /tmp/D_FTA_all_jobstreams_no_dependency.txt

4. Import all the jobs:

       
./<TWS_home>/bin/composer add /tmp/D_FTA_all_jobs.txt

5. Import the job streams without dependency:

       
./<TWS_home>/bin/composer add /tmp/D_FTA_all_jobstreams_no_dependency.txt

6. Import the job streams with dependency:

       
./<TWS_home>/bin/composer replace /tmp/D_FTA_all_jobstreams.txt

7. Delete the job and after, the job streams from source agent. The jobs will start to run on the destination agent (D_FTA) starting with the new plan.

Note: If the jobs are scheduled using time zones it may take 48 hours (2 default plans) for a complete move.

Corrupted FTA Symphony file

To restore the Symphony files when it gets corrupted on any of FTA agent perform the following using the TWS_user:

Method a. (manual restore): 

On the FTA with corrupted Symphony: 
1. Change the limit to 0, to prevent any jobs from beeing launched once the new Symphony file is created: 

       
./<TWS_home>/bin/conman "lc;0"

2. Stop all TWS process on the FTA with corrupted Symphony: 

       
./<TWS_home>/bin/conman "shutdown;wait"

3. Delete / Rename the following files:

       
Symphony, Sinfonia, Jobtable, <TWS_home>/*.msg and <TWS_home>/pobox/*.msg

On the TWS MDM: 
4. Unlink all the agents: 

       
./<TWS_home>/bin/conman "unlink @;noask"

5. Stop the TWS MDM agent: 

       
./<TWS_home>/bin/conman "shutdown;wait"

6. Rename Sinfonia file and copy the Symphony file over the Sinfonia.

 
7. Start the TWS MDM agent: 

       
./<TWS_home>/StartUp

8. Link all the agents: 

       
./<TWS_home>/bin/conman "link @;noask"

On the FTA with corrupted Symphony:  

9. Start the agent: 

       
./<TWS_home>/StartUp

On the TWS MDM: 
10. Start & Link the FTA with corrupted Symphony: 

       
./<TWS_home>/bin/conman "start FTA_name & link FTA_name;noask"

At this stage the new Symphony file should be deployed to the FTA with corrupted Symphony, it will be exactly how it was before the corruption occurred, also as the *msg files were remove / delete some jobs / job stream information may be lost, verify the jobs / job streams status to make sure none of them will be ran 2 times. 

11. Change the limit to the original value: 

       
./<TWS_home>/bin/conman "lc;100"

Method b. (semi-automated restore): 
There a script utility developed by IBM for the corrupted Symphony file that can be found HERE . 

Hint: If there are multiple agents with corrupted Symphony, repeat the above process for each one.

Job distinct / unique / total runs

If a job goes in ABEND and it is not fixed in the same TWS day it will be added to TWS historical database multiple times (as many times as UPDATESTATS jobs run) so the job runs number isn't accurate when we use rep7 to report them, below examples are ran using ORACLE DB.


The below SQL query will remove the duplicate items added by UPDATESTATS for unattended ABEND jobs:

       
alter session set current_schema=maestro_DB_user;

SELECT count(*) FROM (SELECT DISTINCT AJS_NAME, AJB_NAME, JHR_START_TIME, JHR_RUN_DATE, 
JHR_TOTAL_ELAPSED_TIME,JHR_STATUS, WKC_NAME FROM JHR_JOB_HISTORY_RUNS
WHERE jhr_run_date>=trunc(sysdate-12) AND jhr_run_date <trunc(sysdate+1)) 
WHERE (AJB_NAME like 'MAKEPLAN')

Also if you want to see only the unique jobs that ran on a specific workstation:

       
alter session set current_schema=maestro_DB_user;

SELECT count(*) FROM (SELECT UNIQUE AJB_NAME FROM (SELECT DISTINCT AJS_NAME, AJB_NAME, JHR_START_TIME,
JHR_RUN_DATE, JHR_TOTAL_ELAPSED_TIME , JHR_STATUS, WKC_NAME  FROM JHR_JOB_HISTORY_RUNS
WHERE jhr_run_date>=trunc(sysdate-12) AND jhr_run_date <trunc(sysdate+1)) 
WHERE (WKC_NAME like 'TWSMASTER'))

And a nice report containing all job runs, all success runs, all ABEND / CANCEL ones, unique jobs success and unique jobs failed:

       
alter session set current_schema=maestro_DB_user;

SELECT
Count (CASE WHEN JHR_STATUS = 'S' or JHR_STATUS = 'A' 
or JHR_STATUS = 'C' THEN '1' ELSE NULL END) All_Job_Runs,
Count (CASE WHEN JHR_STATUS = 'S' THEN '1' ELSE NULL END) All_Runs_Sucessful,
Count (CASE WHEN JHR_STATUS = 'A' or JHR_STATUS = 'C' THEN '1' ELSE NULL END) All_Runs_Failed,
Count (DISTINCT CASE WHEN JHR_STATUS = 'S' THEN AJB_NAME ELSE NULL END) Unique_Jobs_Successful,
Count (DISTINCT CASE WHEN JHR_STATUS = 'A' 
or JHR_STATUS = 'C' THEN AJB_NAME ELSE NULL END) Unique_Jobs_Failed
FROM (select distinct AJS_NAME, AJB_NAME, JHR_START_TIME, JHR_RUN_DATE, JHR_TOTAL_ELAPSED_TIME,
JHR_STATUS, WKC_NAME from JHR_JOB_HISTORY_RUNS
WHERE jhr_run_date>=trunc(sysdate-12) AND jhr_run_date <trunc(sysdate+1));

Hint: Modify sysdate-+XX to adjust the number of days on which the reports are ran, also you can use dates as: 

       
jhr_run_date>=TO_DATE('10/03/2012','dd/mm/yyyy') AND jhr_run_date <TO_DATE('12/03/2010','dd/mm/yyyy')

SAP Xagent connection test

To check if a SAP Xagent using r3batch is properly configure, run the following command: 

       
./<TWS_home>/methods/r3batch -t PL -c Workstation_name -l "*" -j "*" -- "-debug"

Also additional details can be found in /TWS_home/methods/trace-r3batch.log if you add, for the problematic SAP jobs, "-trace" option.

Hint: To run SAP jobs using TWS you need TWS for Applications to be deployed, which can be installed directly on the TWS master or on on any FTA agent. 

Add a new workstation to plan without waiting for FINAL

To add a new workstation to plan without waiting for FINAL job stream to run use the below command:

./<TWS_home>/JnextPlan -for 0000

The above command will "refresh" the plan, adds new workstation, archives the success jobs (if you dont want to archive the success jobs parameter -noremove should be also used) etc.

Hint: Make sure that all jobs have the CarryForward option to all jobs or the variable enCarryForward set to ALL. 

rep7 SQL query

rep7 is very slow and hard to use it, below is the SQL queries version for MAKEPLAN job (using ORACLE DB):

       
alter session set current_schema=maestro_DB_user;

SELECT DISTINCT AJS_NAME, AJB_NAME, JHR_START_TIME, JHR_RUN_DATE, JHR_TOTAL_ELAPSED_TIME, 
JHR_STATUS, WKC_NAME FROM JHR_JOB_HISTORY_RUNS
WHERE (AJB_NAME like 'MAKEPLAN') order by AJB_NAME

Job average run

The below SQL query calculates the job average run time (in our case MAKEPLAN) using the historical data stored in the DB (using ORACLE DB):

       
alter session set current_schema=maestro_DB_user;

select avg (JHR_TOTAL_ELAPSED_TIME) as average
FROM JHR_JOB_HISTORY_RUNS
where AJB_NAME like 'MAKEPLAN'

If you want to calculate the average run time only for success jobs run:

       
alter session set current_schema=maestro_DB_user;

select avg (JHR_TOTAL_ELAPSED_TIME) as average
FROM JHR_JOB_HISTORY_RUNS
where AJB_NAME like 'MAKEPLAN' and JHR_STATUS = 'S'

Hint: The historical data age is controlled by the variable statsHistory / sh, the default value is 10 days.

Jobs without Job Stream

To find out which Jobs does not belong to Job Stream, the easiest way is to run a SQL query (using ORACLE DB):

       
alter session set current_schema=maestro_DB_user;

SELECT JOD_NAME FROM JOD_JOB_DEFINITIONS
WHERE JOD_NAME NOT IN
(SELECT AJB_NAME FROM AJB_ABSTRACT_JOBS)

Security File Corrupted

In some rare cases the TWS Security file gets corrupted, you will receive the below error message, which is also found in SystemOut.log file:

AWSDEJ027E Either the user for which authorization is being sought is not defined  n the Security file, or it is defined but no associated access rights are defined. The Security file is probably either empty or corrupt.

To resolve the problem perform the below steps:

       
1.  login as root
2.  . ./<TWS_home>/tws_env.sh
3.  rm Security
4.  ./<TWS_home>/bin/makesec Security.conf




After applying the above steppes the "default" Security file is restored and at this time you can apply Security file back-up created before:




       
./<TWS_home>/bin/makesec Security_bck




Hint: Always do a Security file back-up every time you perform any changes on it, i do the back-ups as Security_<date>, e.g. Security_20120229.