Master (MDM or DM) switch (promotion)

In case of any failure on primary master (MDM or DM) TWS has the option to promote a new master in real time.

There are 2 cases that we will go thru:
          1) Temporary switch (the new master will act as a primary master until the FINAL job stream is set to run).
          2) Permanent switch (the new master will act as a primary master until a switchmgr command is performed).

1) Temporary switch
The new promoted master will act as a primary master until the FINAL job stream runs after, it will be reverted back to "old" primary master.
To make the switch / promotion run on the target (the command is instant and you will see in CPU screen at "NODE" column the new "MASTER"):

./<TWS_home>/switchmgr

!!! Applicable to any master or to any FTA which has in it's "TYPE" definition column "FULLSTATUS ON"

2) Permanent switch
The new master will act as a primary master until a switchmgr command should be performed on another master only (Back-up, DR etc). 

a) Cancel the FINAL job stream from the current plan.

./<TWS_home>/bin/conman "cancel @#final;noask"

b) Edit and update the FINAL job stream so it can run on the new primary master.

./<TWS_home>/bin/composer "mo @#final;full"

!!! NOTE change the Job Stream Workstation, Job Workstation and the script path

c) Submit to plan the new "FINAL" job stream

./<TWS_home>/bin/conman "sbs @#final;noask"

!!! NOTE If the "FINAL" job stream update was performed using CLI (as above) MAKE sure you will be deleting the "original" "FINAL" job stream. CLI is not replacing the objects is the workstation is change, instead it creates another item

d) Change in the workstation definition the "TYPE" option of the current master from "MANAGER" to fault-tolerant agent "FTA". 

./<TWS_home>/bin/composer "mo cpu <current master>" 

CPUNAME <current master> 
  DESCRIPTION "MASTER CPU"
  OS UNIX
  NODE master.tws.com TCPADDR 31234
  DOMAIN MASTERDM
  FOR MAESTRO
    TYPE MANAGER #change it to FTA
    AUTOLINK ON
    BEHINDFIREWALL OFF
    FULLSTATUS ON
END

e) Change in the workstation definition the "TYPE" option of the master or the fault-tolerant agent "FTA" which will be promoted to primary master to "MANAGER".

./<TWS_home>/bin/composer "mo cpu <future master>" 

CPUNAME <future master>
  DESCRIPTION "BACK-up MASTER CPU"
  OS UNIX
  NODE back-up-master.tws.com TCPADDR 31234
  DOMAIN MASTERDM
  FOR MAESTRO
    TYPE FTA #change it to MANAGER
    AUTOLINK ON
    BEHINDFIREWALL OFF
    FULLSTATUS ON
END

f) Copy the Security file from the <current master> to the <future master>, by doing this we want to make sure the users will have the same rights.

g) Use "switchmgr" function on <future master> to switch the <future master> as the current master.

./<TWS_home>/switchmgr

Optional a) Use "switcheventprocessor" on <future master> to switch the event rules processing server to the <future master> master (only if the event rules are being used and are active on <current master>).

./<TWS_home>/conman switcheventprocessor

Optional b) On the <future master>, edit the "globalopts" file and change the master option to the name of the <future master> workstation (this is used mainly for reports).

vi /<TWA_home>/TWS/mozart/globalopts 

# TWS globalopts file on the TWS Master defines attributes
# of the TWS network.
#
#----------------------------------------------------------------------------
company                     =TWS
master=<current master> # change it to <future master>
start                       =0000

!!! IMPORTANT: The multiple-masters domain (e.g. back-up / DR masters and/or "FULLSTATUS ON" FTA's) global settings and its behavior.



Command line (CLI) environment setup

To set up the command line output (conman, composer) there are 3 variables that can be adjusted to obtain the optimum output format.

MAESTROLINES: Specifies the number of lines per screen. At the end of each screen page, conman prompts to continue. If MAESTROLINES (or LINES) is set to zero or a negative number, conman does not pause at the end of a page. 

MAESTROCOLUMNS: Specifies the number of characters per line. The following options are available: Less than 120 (only some columns will be displayed). Equal to or more than 120 (all columns will be displayed). The maximum value is 1024. If either MAESTROLINES or MAESTROCOLUMNS is not set, the standard shell variables, LINES and COLUMNS, are used. 

MAESTRO_OUTPUT_STYLE: Specifies how object names are displayed. If set to LONG, full names are displayed. If not set, or set to any value other than LONG, long names are truncated to eight characters followed by a plus sign (+). 

Default unix output: 

       
bash-3.00$ ./<TWS_home>/bin/conman sj @#final
Tivoli Workload Scheduler (UNIX)/CONMAN 8.6.0.00 (20110719)
Licensed Materials - Property of IBM*
5698-WSH
(C) Copyright IBM Corp. 1998, 2011 All rights reserved.
* Trademark of International Business Machines
Installed for user "maestro".
Locale LANG set to the following: "en"
Scheduled for (Exp) 04/04/12 (#89) on TW1T.  Batchman LIVES.  Limit: 20, Fence: 0, Audit Level: 1
%sj @#final
                                               (Est)  (Est)
CPU      Schedule SchedTime  Job      State Pr Start  Elapse RetCode Deps

TW1T    #FINAL    0003 04/04 ******** SUCC  10 00:03  00:01          [04/03/12]
                             STARTAP+ SUCC  10 00:03  00:01        0 #J18294
                             MAKEPLAN SUCC  10 00:03  00:01        0 #J18468
                             SWITCHP+ SUCC  10 00:03  00:01        0 #J18583
                             CREATEP+ SUCC  10 00:04  00:01        0 #J18990
                             UPDATES+ SUCC  10 00:04  00:01        0 #J18996

TW1T    #FINAL    0003 04/05 ******** HOLD  10(04/05)(00:05)         [Carry]
                             STARTAP+ HOLD  10(04/05)(00:01)
                             MAKEPLAN HOLD  10(04/05)(00:01)         STARTAP+
                             SWITCHP+ HOLD  10(04/05)(00:01)         MAKEPLAN
                             CREATEP+ HOLD  10(04/05)(00:01)         SWITCHP+
                             UPDATES+ HOLD  10(04/05)(00:01)         SWITCHP+

And the output with variables set as: 
MAESTRO_OUTPUT_STYLE=LONG
MAESTROLINES=0
MAESTROCOLUMS=512

       
bash-3.00$ ./<TWS_home>/bin/conman sj @#final
Tivoli Workload Scheduler (UNIX)/CONMAN 8.6.0.00 (20110719)
Licensed Materials - Property of IBM*
5698-WSH
(C) Copyright IBM Corp. 1998, 2011 All rights reserved.
* Trademark of International Business Machines
Installed for user "maestro".
Locale LANG set to the following: "en"
Scheduled for (Exp) 04/04/12 (#89) on TW1T.  Batchman LIVES.  Limit: 20, Fence: 0, Audit Level: 1
%sj @#@final
                                                                                               (Est)  (Est)
Workstation      Job Stream       SchedTime  Job                                      State Pr Start  Elapse ReturnCode Dependencies

TW1T            #FINAL            0003 04/04 **************************************** SUCC  10 00:03  00:01             [04/03/12]
                                             STARTAPPSERVER                           SUCC  10 00:03  00:01           0 #J18294
                                             MAKEPLAN                                 SUCC  10 00:03  00:01           0 #J18468
                                             SWITCHPLAN                               SUCC  10 00:03  00:01           0 #J18583
                                             CREATEPOSTREPORTS                        SUCC  10 00:04  00:01           0 #J18990
                                             UPDATESTATS                              SUCC  10 00:04  00:01           0 #J18996

TW1T            #FINAL            0003 04/05 **************************************** HOLD  10(04/05)(00:05)            [Carry]
                                             STARTAPPSERVER                           HOLD  10(04/05)(00:01)
                                             MAKEPLAN                                 HOLD  10(04/05)(00:01)            STARTAPPSERVER
                                             SWITCHPLAN                               HOLD  10(04/05)(00:01)            MAKEPLAN
                                             CREATEPOSTREPORTS                        HOLD  10(04/05)(00:01)            SWITCHPLAN
                                             UPDATESTATS                              HOLD  10(04/05)(00:01)            SWITCHPLAN

To have the variables set by default, add the following lines to your user profile file (.profile, .bash_profile etc): 

       
<TWS_home>/tws_env.sh
export MAESTRO_OUTPUT_STYLE=LONG
export MAESTROLINES=0
export MAESTROCOLUMS=512

FTA Stop & Start commands & script

To Start & Stop a TWS FTA use the below commands: 

1. Start a TWS FTA:

       
./<TWS_home>/StartUp
./<TWS_home>/bin/conman "start & link @;noask"

2. Stop a TWS FTA:

       
./<TWS_home>/bin/conman "shutdown;wait"

And a small script to start / stop TWS FTA:

       
#!/bin/sh
#
# Start/Stop the Tivoli Workload Scheduler FTA
#

case "$1" in

   start)
        /<TWS_home>/StartUp
        /<TWS_home>/bin/conman "start & link @;noask"
        ;;
   stop)
        /<TWS_home>/bin/conman "shutdown;wait"
        ;;
   *)
      echo "Unknown flag passed to $0."
      echo "Usage: $0 {start, stop}"

esac
exit 0

Note: The above script is the simplest possible, I like simple things, even we have a error as it will try to start / stop all the agents but it does not have rights, which is normal.