Multiple-masters domain enSwFaultTol set-up

The multiple-masters domain (e.g. containing back-up / DR masters and/or "FULLSTATUS ON" FTA's) biggest challenge is to make sure there is no loss on jobs details (messages) in case the primary master is out of order and we need to promote another master or "FULLSTATUS ON" FTA's to primary master.

There is a global variable called: enSwFaultTol enable or disable the fault-tolerant switch manager feature which basically does changes the flow of communication inside TWS environment:

 enSwFaultTol / sw = NO The FTA's are connected and are sending the jobs data to primary master and other masters and / or "FULLSTATUS ON" FTA's are getting in sync with primary master.

enSwFaultTol / sw = YES The FTA's are connected and are sending the jobs data to each master and / or "FULLSTATUS ON" FTA's independently.

Below is  a diagram representing only the communication channels that are impacted by the above variable.

Default is NO but my recommendation is to change it to YES, even if IBM recommends NO (I've had several meetings with IBM support on this topic but they did not convince me to use it as NO).

Why?

• enSwFaultTol / sw = NO biggest downsize is that in case of primary master being in a freeze state you will lose all the jobs messages sent to it by FTA's until the new primary master promotion. It happened to me several times and we lost up to several hours worth of data. 
• enSwFaultTol / sw = YES the above scenario won't happen as all the other masters or "FULLSTATUS ON" FTA's are receiving all the jobs messages but its small disadvantage is an increase network traffic as the data is sent to multiple servers. 

 My strong recommendation, even is it against IBM one, is to use enSwFaultTol / sw = YES on a multiple-masters domain.

Master (MDM or DM) switch (promotion)

In case of any failure on primary master (MDM or DM) TWS has the option to promote a new master in real time.

There are 2 cases that we will go thru:
          1) Temporary switch (the new master will act as a primary master until the FINAL job stream is set to run).
          2) Permanent switch (the new master will act as a primary master until a switchmgr command is performed).

1) Temporary switch
The new promoted master will act as a primary master until the FINAL job stream runs after, it will be reverted back to "old" primary master.
To make the switch / promotion run on the target (the command is instant and you will see in CPU screen at "NODE" column the new "MASTER"):

./<TWS_home>/switchmgr

!!! Applicable to any master or to any FTA which has in it's "TYPE" definition column "FULLSTATUS ON"

2) Permanent switch
The new master will act as a primary master until a switchmgr command should be performed on another master only (Back-up, DR etc). 

a) Cancel the FINAL job stream from the current plan.

./<TWS_home>/bin/conman "cancel @#final;noask"

b) Edit and update the FINAL job stream so it can run on the new primary master.

./<TWS_home>/bin/composer "mo @#final;full"

!!! NOTE change the Job Stream Workstation, Job Workstation and the script path

c) Submit to plan the new "FINAL" job stream

./<TWS_home>/bin/conman "sbs @#final;noask"

!!! NOTE If the "FINAL" job stream update was performed using CLI (as above) MAKE sure you will be deleting the "original" "FINAL" job stream. CLI is not replacing the objects is the workstation is change, instead it creates another item

d) Change in the workstation definition the "TYPE" option of the current master from "MANAGER" to fault-tolerant agent "FTA". 

./<TWS_home>/bin/composer "mo cpu <current master>" 

CPUNAME <current master> 
  DESCRIPTION "MASTER CPU"
  OS UNIX
  NODE master.tws.com TCPADDR 31234
  DOMAIN MASTERDM
  FOR MAESTRO
    TYPE MANAGER #change it to FTA
    AUTOLINK ON
    BEHINDFIREWALL OFF
    FULLSTATUS ON
END

e) Change in the workstation definition the "TYPE" option of the master or the fault-tolerant agent "FTA" which will be promoted to primary master to "MANAGER".

./<TWS_home>/bin/composer "mo cpu <future master>" 

CPUNAME <future master>
  DESCRIPTION "BACK-up MASTER CPU"
  OS UNIX
  NODE back-up-master.tws.com TCPADDR 31234
  DOMAIN MASTERDM
  FOR MAESTRO
    TYPE FTA #change it to MANAGER
    AUTOLINK ON
    BEHINDFIREWALL OFF
    FULLSTATUS ON
END

f) Copy the Security file from the <current master> to the <future master>, by doing this we want to make sure the users will have the same rights.

g) Use "switchmgr" function on <future master> to switch the <future master> as the current master.

./<TWS_home>/switchmgr

Optional a) Use "switcheventprocessor" on <future master> to switch the event rules processing server to the <future master> master (only if the event rules are being used and are active on <current master>).

./<TWS_home>/conman switcheventprocessor

Optional b) On the <future master>, edit the "globalopts" file and change the master option to the name of the <future master> workstation (this is used mainly for reports).

vi /<TWA_home>/TWS/mozart/globalopts 

# TWS globalopts file on the TWS Master defines attributes
# of the TWS network.
#
#----------------------------------------------------------------------------
company                     =TWS
master=<current master> # change it to <future master>
start                       =0000

!!! IMPORTANT: The multiple-masters domain (e.g. back-up / DR masters and/or "FULLSTATUS ON" FTA's) global settings and its behavior.



Command line (CLI) environment setup

To set up the command line output (conman, composer) there are 3 variables that can be adjusted to obtain the optimum output format.

MAESTROLINES: Specifies the number of lines per screen. At the end of each screen page, conman prompts to continue. If MAESTROLINES (or LINES) is set to zero or a negative number, conman does not pause at the end of a page. 

MAESTROCOLUMNS: Specifies the number of characters per line. The following options are available: Less than 120 (only some columns will be displayed). Equal to or more than 120 (all columns will be displayed). The maximum value is 1024. If either MAESTROLINES or MAESTROCOLUMNS is not set, the standard shell variables, LINES and COLUMNS, are used. 

MAESTRO_OUTPUT_STYLE: Specifies how object names are displayed. If set to LONG, full names are displayed. If not set, or set to any value other than LONG, long names are truncated to eight characters followed by a plus sign (+). 

Default unix output: 

       
bash-3.00$ ./<TWS_home>/bin/conman sj @#final
Tivoli Workload Scheduler (UNIX)/CONMAN 8.6.0.00 (20110719)
Licensed Materials - Property of IBM*
5698-WSH
(C) Copyright IBM Corp. 1998, 2011 All rights reserved.
* Trademark of International Business Machines
Installed for user "maestro".
Locale LANG set to the following: "en"
Scheduled for (Exp) 04/04/12 (#89) on TW1T.  Batchman LIVES.  Limit: 20, Fence: 0, Audit Level: 1
%sj @#final
                                               (Est)  (Est)
CPU      Schedule SchedTime  Job      State Pr Start  Elapse RetCode Deps

TW1T    #FINAL    0003 04/04 ******** SUCC  10 00:03  00:01          [04/03/12]
                             STARTAP+ SUCC  10 00:03  00:01        0 #J18294
                             MAKEPLAN SUCC  10 00:03  00:01        0 #J18468
                             SWITCHP+ SUCC  10 00:03  00:01        0 #J18583
                             CREATEP+ SUCC  10 00:04  00:01        0 #J18990
                             UPDATES+ SUCC  10 00:04  00:01        0 #J18996

TW1T    #FINAL    0003 04/05 ******** HOLD  10(04/05)(00:05)         [Carry]
                             STARTAP+ HOLD  10(04/05)(00:01)
                             MAKEPLAN HOLD  10(04/05)(00:01)         STARTAP+
                             SWITCHP+ HOLD  10(04/05)(00:01)         MAKEPLAN
                             CREATEP+ HOLD  10(04/05)(00:01)         SWITCHP+
                             UPDATES+ HOLD  10(04/05)(00:01)         SWITCHP+

And the output with variables set as: 
MAESTRO_OUTPUT_STYLE=LONG
MAESTROLINES=0
MAESTROCOLUMS=512

       
bash-3.00$ ./<TWS_home>/bin/conman sj @#final
Tivoli Workload Scheduler (UNIX)/CONMAN 8.6.0.00 (20110719)
Licensed Materials - Property of IBM*
5698-WSH
(C) Copyright IBM Corp. 1998, 2011 All rights reserved.
* Trademark of International Business Machines
Installed for user "maestro".
Locale LANG set to the following: "en"
Scheduled for (Exp) 04/04/12 (#89) on TW1T.  Batchman LIVES.  Limit: 20, Fence: 0, Audit Level: 1
%sj @#@final
                                                                                               (Est)  (Est)
Workstation      Job Stream       SchedTime  Job                                      State Pr Start  Elapse ReturnCode Dependencies

TW1T            #FINAL            0003 04/04 **************************************** SUCC  10 00:03  00:01             [04/03/12]
                                             STARTAPPSERVER                           SUCC  10 00:03  00:01           0 #J18294
                                             MAKEPLAN                                 SUCC  10 00:03  00:01           0 #J18468
                                             SWITCHPLAN                               SUCC  10 00:03  00:01           0 #J18583
                                             CREATEPOSTREPORTS                        SUCC  10 00:04  00:01           0 #J18990
                                             UPDATESTATS                              SUCC  10 00:04  00:01           0 #J18996

TW1T            #FINAL            0003 04/05 **************************************** HOLD  10(04/05)(00:05)            [Carry]
                                             STARTAPPSERVER                           HOLD  10(04/05)(00:01)
                                             MAKEPLAN                                 HOLD  10(04/05)(00:01)            STARTAPPSERVER
                                             SWITCHPLAN                               HOLD  10(04/05)(00:01)            MAKEPLAN
                                             CREATEPOSTREPORTS                        HOLD  10(04/05)(00:01)            SWITCHPLAN
                                             UPDATESTATS                              HOLD  10(04/05)(00:01)            SWITCHPLAN

To have the variables set by default, add the following lines to your user profile file (.profile, .bash_profile etc): 

       
<TWS_home>/tws_env.sh
export MAESTRO_OUTPUT_STYLE=LONG
export MAESTROLINES=0
export MAESTROCOLUMS=512