Brocade Fiber Channel Switch - CLI Zoning

May 13, 2020 by Abe Covello in Infratsructure, Network

Man, I hate Java. Absolutely hate it. Especially when you’re trying to connect to an old, outdated, and non-updated device like a Brocade 300 FC switch.

Sure, I could dig up whatever outdated Java install the switch is calling for, install it, troubleshoot for an hour why Firefox still won’t load the GUI, start drinking heavily, give up on life, and become a hermit. But that seems a little extreme just because of a switch. So instead, I’ll zone my switch through the CLI, which I haven’t done in YEARS. So lets get started!

In this case, I’m connecting three new ESXi hosts to an existing pair of FC switches, with a SAN and several hosts already setup. Start by connecting the hosts to your FC switch. In my case I have dual-port FC cards in the hosts and two FC switches, so I’m going to connect one port from each host to one switch. Make note of where you plug your hosts into, and where the SAN is connected. In my case it looks something like this:

In this case, our FC switches are not stacked/connected to ensure separate pathing, so we’ll need to configure each switch separately.

Once the new hosts are connected to the FC switches, SSH into the first switch as an admin user and run cfgshow to display the current zoning configuration.

brcdsw01:admin> cfgshow
Defined configuration:
cfg: Production
ESXi01_Zone; ESXi02_Zone;ESXi03_Zone; ESXi04_Zone
zone: ESXi01_Zone
SAN_0A; SAN_1A; ESXi01
zone: ESXi02_Zone
SAN_0A; SAN_1A; ESXi02
zone: ESXi03_Zone
SAN_0A;SAN_1A; ESXi03
zone: ESXi04_Zone
SAN_0A; SAN_1A; ESXi04
alias: SAN_0A
50:00:00:00:00:00:00:30; 50:00:00:00:00:00:00:30
alias: SAN_1A
50:00:00:00:00:00:00:38; 50:00:00:00:00:00:00:38
alias: ESXi01 10:00:00:00:00:00:00:d5; 20:00:00:00:00:00:00:d5
alias: ESXi02 10:00:00:00:00:00:00:5d; 20:00:00:00:00:00:00:5d
alias: ESXi03 10:00:00:00:00:00:00:00; 20:00:00:00:00:00:00:93
alias: ESXi04 10:00:00:00:00:00:00:46; 20:00:00:00:00:00:00:46

Effective configuration:
cfg: Production
zone: ESXi01_Zone
50:00:00:00:00:00:00:30
50:00:00:00:00:00:00:38
10:00:00:00:00:00:00:d5
20:00:00:00:00:00:00:d5
zone: ESXi02_Zone
50:00:00:00:00:00:00:30
50:00:00:00:00:00:00:38
10:00:00:00:00:00:00:5d
20:00:00:00:00:00:00:5d
zone: ESXi03_Zone
50:00:00:00:00:00:00:30
50:00:00:00:00:00:00:38
10:00:00:00:00:00:00:93
20:00:00:00:00:00:00:93
zone: ESXi04_Zone
50:00:00:00:00:00:00:30
50:00:00:00:00:00:00:38
10:00:00:00:00:00:00:46
20:00:00:00:00:00:00:46

Without going into too much detail, the above output tells us the current switch config has a Configuration Group called “Production”. That Group has four Zones: ESXi01_Zone; ESXi02_Zone;ESXi03_Zone; ESXi04_Zone. Each one of those Zones contains three aliases: one entry for the hosts and two entries for the SAN (Redundant SAN controllers, side A and side B). Each Alis has the discovered WWPNs for each attached system. To configure the switches for our new hosts, we’ll be working backwards, creating an alias for each hosts, then putting those aliases into zones, then updating and enabling our Configuration Group.

Run the command switchshow to see the port state and WWPN discovered on each port.

brcdsw01:admin> switchshow
switchName: brcdsw01
switchType: 71.2
switchState: Online
switchMode: Native
switchRole: Principal
switchDomain: 20
switchId: fffc14
switchWwn: 10:00:00:05:33:2d:57:b4
zoning: ON (Production)
switchBeacon: OFF
HIF Mode: OFF

Index Port Address Media Speed State Proto
==================================================

0 0 140000 id N8 Online FC F-Port 50:00:00:00:00:00:00:30
1 1 140100 id N8 Online FC F-Port 50:00:00:00:00:00:00:38
2 2 140200 id N8 No_Light FC
3 3 140300 id N8 No_Light FC
4 4 140400 id N8 Online FC F-Port 21:00:00:00:00:00:00:4c
5 5 140500 id N8 Online FC F-Port 21:00:00:00:00:00:00:54
6 6 140600 id N8 Online FC F-Port 21:00:00:00:00:00:00:60

***Additional port config removed***

Ports 0 and 1 are the connections to the SAN, and ports 4, 5, & 6 connect to each of the new hosts.

Lets create the new aliases for each of the three new hosts:

brcdsw01:admin> alicreate "LabHost01", "21:00:00:00:00:00:00:4c"
brcdsw01:admin> alicreate "LabHost02", "21:00:00:00:00:00:00:54"
brcdsw01:admin> alicreate "LabHost03", "21:00:00:00:00:00:00:60"

We can rerun the cfgshow command to verify our aliases have been created:

brcdsw01:admin> cfgshow
Defined configuration:
*****************************************
alias: LabHost01
21:00:00:00:00:00:00:4c
alias: LabHost02
21:00:00:00:00:00:00:54
alias: LabHost03
21:00:00:00:00:00:00:60

Now we’ll need to create three new zones containing one host alias, and the two SAN aliases, again using cfgshow to verify our new Zones:

brcdsw01:admin> zonecreate "LabHost01_Zone", "LabHost01; SAN_0A; SAN_1A"
brcdsw01:admin> zonecreate "LabHost02_Zone", "LabHost02; SAN_0A; SAN_1A"
brcdsw01:admin> zonecreate "LabHost03_Zone", "LabHost03; SAN_0A; SAN_1A"

brcdsw01:admin> cfgshow
Defined configuration:
*****************************************
zone: LabHost01_Zone
LabHost01; SAN_0A; SAN_1A
zone: LabHost02_Zone
LabHost02; SAN_0A; SAN_1A
zone: LabHost03_Zone
LabHost03; SAN_0A; SAN_1A

***********************************************
alias: LabHost01
21:00:00:00:00:00:00:4c
alias: LabHost02
21:00:00:00:00:00:00:54
alias: LabHost02
21:00:00:00:00:00:00:60

Now we need to add the new zones to existing Configuration Group:

If you’re setting the zoning up from scratch, use the command cfgcreate.

brcdsw01:admin> cfgadd "Production", "LabHost01_Zone"
brcdsw01:admin> cfgadd "Production", "LabHost02_Zone"
brcdsw01:admin> cfgadd "Production", "LabHost03_Zone"

brcdsw01:admin> cfgshow
Defined configuration:
cfg: Production
ESXi01_Zone; ESXi02_Zone; ESXi03_Zone; ESXi04_Zone; LabHost01_Zone; LabHost02Zone; LabHost03Zone

If you don’t already have a Configuration Group, use the command:

cfgcreate “CONFIG_NAME”, “ZONE_NAME”

Now that we have updated out configuration, we finally need to save and commit the changes for them to become active. Use the commands cfgsave and cfgenable to save and commit the changes we have just made:

brcdsw01:admin> cfgsave
WARNING!!!
The changes you are attempting to save will render the
Effective configuration and the Defined configuration
inconsistent. The inconsistency will result in different
Effective Zoning configurations for switches in the fabric if
a zone merge or HA failover happens. To avoid inconsistency
it is recommended to commit the configurations using the
'cfgenable' command.

Do you want to proceed with saving the Defined
zoning configuration only? (yes, y, no, n): [no] yes
Updating flash ...
brcdsw01:admin> cfgenable "Production"
You are about to enable a new zoning configuration.
This action will replace the old zoning configuration with the
current configuration selected. If the update includes changes
to one or more traffic isolation zones, the update may result in
localized disruption to traffic on ports associated with
the traffic isolation zone changes
Do you want to enable 'Production' configuration (yes, y, no, n): [no] yes
zone config "Production" is in effect
Updating flash ...

If we take a look at cfgshow one last time, we’ll see the Effective Configuration now has our new zones:

brcdsw01:admin> cfgshow
****************************
Effective configuration:
cfg: Production
**************************
zone: LabHost01_Zone
21:00:00:00:00:00:00:4c
50:00:00:00:00:00:00:30
50:00:00:00:00:00:00:38
zone: LabHost02_Zone
21:00:00:00:00:00:00:54
50:00:00:00:00:00:00:30
50:00:00:00:00:00:00:38
zone: LabHost02_Zone
21:00:00:00:00:00:00:60
50:00:00:00:00:00:00:30
50:00:00:00:00:00:00:38

Now rinse and repeat for your second FC switch if necessary.

Once both switches are zoned, you can now configure your SAN to allow LUN access to your new hosts.

Mobile AP solution - Ruckus M510 AP

April 24, 2019 by Abe Covello in Wireless, Network

I have a prospective customer who, during the course of our conversation, explained that one of their issues was for mobile service workers to have wireless connectivity back into their network. Using a Ruckus SmartZone SZ-100 AP controller and GRE tunneling from the APs, this can be accomplished with relative ease. Or so the Ruckus marketing materials and my local Ruckus SE led me to believe. It ended up being easy, but the documentation and knowledge was lacking so it took quite a bit of trial and error on my part to get it figured out.

The M510 AP from Ruckus is an indoor AP designed for LTE backhaul, which makes it perfect for deployment in vehicles, pop-up locations, and temporary office locations such as construction sites or first-responder field operations.

Ruckus M510 Data Sheet

My SmartZone is running as a VM without a Data Plane, so I can’t run GRE tunneling back to my controller. This works for my demo purpose, but doesn’t give my clients an experience as if they were inside the network. For a production environment I’d suggest either a vSZ/vSZ-D deployment, or a SZ-100 physical appliance and use tunneling from the AP to the controller.

The first thing you’ll need to do is contact AT&T and get a SIM card with a data plan. Make sure you get a Micro-SIM card. The M510 can take up to two SIM cards, but currently only works with AT&T. I believe that another carrier (Sprint?) is being tested and should be available sometime this year but it’s only AT&T for now, sorry. Also, you’ll need a M510 AP and an AP license for your Smart Zone.

AT&T Compatible Data Plan

Before you get into the SmartZone configuration change, plan out how your mobile wireless will be utilized and deployed. If you are going to broadcast your internal SSIDs through your mobile APs, then you can get away with just your Default AP and Wireless Zone and just worry about creating the appropriate AP Groups. Becasue I’m going to be broadcasting an SSID only for testing and demo purposes, I’ll create separate AP and Wireless Zones/Groups for my M510 AP.

Setup SmartZone for External Access

First things first, for this to work you’re going to need to get your SmartZone an external IP address and set up some firewall rules. The ports you’ll need to forward are:

TCP ports:
Port 21 = FTP Com port
Port range 16384-65000 (configurable in vSCG) = FTP Passive port range
Port 443 = https port for AP to vSCG Registration
Port 22 = SSH Tunnel from AP to SCG
Port 91 = AP and vSCG firmware update and other uses port
Optional TCP Ports:
Port 8443 = Used for vSCG WebUI access from remote (if needed, this may not be necessary if you are using a 3 interface configuration and using VPN to connect to management interface)
Port 7443 = Port used by SWIPE (not required if you are not using SWIPE to provision APs onsite)

UDP Ports:
Port 12223 = LWAP communication port for APs not running vSCG firmware to communicate with vSCG and get upgraded (using passive FTP ports mentioned above

You also need to set the external NAT address of your SmartZone so that the APs keep the correct IP address for their SCG connections. Go to System/Cluster and edit your Control Plane Network Settings. Set the Control NAT IP Address to your external IP address and click save.

Setup AP Zones and Groups

Lets start by setting up an AP group that the M510 AP(s) will live in. Since we’ll want to have different settings for our mobile APs, it’s easiest to create a new AP group to manage these settings instead of making changes to each individual AP. Because this is a demo environment, I’m also going to create an AP Zone. I want to have obvious delination of my production and demo environment only setting up one AP Zone, and a single AP Group. If this was a production environment, I’d have multiple AP groups depending on how my mobile APs were being deployed (AP per site, or AP per function). I like to keep my Zones and groups in a parent-child config, so all of my general mobile AP settings will be set in my AP Zone. Any site or function specific settings will be configured in the AP Group.

I have created an AP Zone called MobileZone, and I’ll set my AP model specific settings in this zone.

Since you’ll be using AT&T for your LTE backhaul, contact them to make sure the APN is correct, but for me it’s “broadband”. I set my M510 COnfiguration settings to mostly default, except for the primary and secondary APN, and the LAN 2 port is set to “Default Trunk Port (LAN)” instead of WAN. This will let me plug a switch into the port and connect additional devices via ethernet. I also set my WAN connection to “Cellular (Primary) with Ethernet failover”. Make any other AP settings you want here, such as channel selection, AP reboot timeout, or radio channels and transmit power.

If you;’re deploying multiple APs and need different AP Groups, configure those now and make any AP Group setting changes you want. Again, since this is a demo setup, I’m using a single default AP group with default AP settings.

If you have added your M510 to your SmartZone, you can now move it into the correct Zone and group.

Setup Wireless Zones and Groups

When you create a Zone and a Group either in the AP configuration or the Wireless LAN configuration, they get created in both locations. So since we created an AP Zone and Group earlier, we’ll see them in our Wireless LAN configuration as well. So now we can create our new wireless network for our mobile APs.

Create your new wireless network and configure it as needed. Don’t forget to go back to your Wireless LAN Group configuration and add your new WLAN.

Also make sure that in your AP Group you have the correct WLAN group selected for broadcast.

At this point your M510 AP should be broadcasting your SSID and connecting to your AT&T LTE. The M510 AP will attempt to connect to the network through the assigned WAN interface if a cable is plugged in, so if you’re planning on using PoE to power the AP, make sure to set the PoE interface as your LAN connection (instead of the default WAN). I suggest powering it through an AC adaptor or the built-in DC 9-16V power block.

GPS Location and Map Display

In order to get SmartZone to show your M510 location you will need to get a Google Maps API key. From the Dashboard, click on the Map settings icon.

This opens the Map Settings window. Click on the Google Map API Key and enter your API key. If you don’t have an API key, click on the link shown in the window and get an API key.

It took a while the first time, but eventually my M510 AP started showing up on the map.

DHCP

Probably the easiest way to deploy DHCP out to your mobile sites is to put a DHCP server on your M510 APs. The SmartZone allows you to run a DHCP server on each AP, run DHCP across multiple APs, or run DHCP on a hierarchical AP setup.

Regardless on how you pan on implementing DHCP on the APs, the setup is the same. In your SmartZone, go to Services & Profiles / DHCP & NAT. The first thing we need to do is create our DHCP Pool(s). Click the DHCP Pool tab, and then create a new DHCP pool in the correct Zone (MobileZone in my case).

Once the DHCP Pool is created, go to the DHCP Settings tab. This is where we’ll define the type of DHCP hierarchy we’ll be running. Click on your Zone and then click the Enable DHCP Services on AP button.

Click Edit DHCP Serveries on AP button and follow the prompts to pick your type of DHCP hierarchy, the Pool associated with your server, and the APs you want to run DHCP services on.

You should now have a fully configured and running Ruckus M510 AP!

Search Avamar backups for a file or folder

January 17, 2019 by Abe Covello in Avamar

Do you get requests to restore a file or folder from Avamar only to have the user have no idea where it was actually located? Do you not have Data Protection Search?

Well, here is a fairly easy way to get the location you need from Avamar CLI.

avtar --list --account=path_of_client --after="YYYY-MM-DD hh:mm:ss" --before="YYYY-MM-DD hh:mm:ss" --verbose | grep file_name

where

path_of_client is the Avamar domain location of the system /clients/servername

file_name is a file or folder name you want to find.

Convert MTree to DDBoost Storage Unit

July 26, 2018 by Abe Covello in Data Domain

If you're migrating from one Data Domain to another and using a non-Avamar backup solution (Veeam, etc) and the built-in DD replication, you'll need to create a new storage unit post replication. Data Domain won't let you create a new SU from the GUI from the replicated MTree so, you'll need to convert the replicated MTree to a DDBoost SU from the CLI:

sysadmin@DDxxxx# mtree list
Name                           Pre-Comp (GiB)   Status
----------------------------   --------------   ------
/data/col1/REPLICATED_MTREE            1924.5   RW
/data/col1/backup                         0.0   RW
----------------------------   --------------   ------
 D    : Deleted
 Q    : Quota Defined
 RO   : Read Only
 RW   : Read Write
 RD   : Replication Destination
 RLGE : Retention-Lock Governance Enabled
 RLGD : Retention-Lock Governance Disabled
 RLCE : Retention-Lock Compliance Enabled

We see our replicated MTree directory REPLICATED_MTREE, and we'll now turn that into a DDBoost storage unit with:

sysadmin@DDxxxx# ddboost storage-unit modify REPLICATED_MTREE user ddboost
Storage-unit "REPLICATED_MTREE" modified for user "ddboost".

Verify the storage unit with:

sysadmin@DDxxxx# ddboost storage-unit show
Name                Pre-Comp (GiB)   Status   User      Report Physical
                                                           Size (MiB)
-----------------   --------------   ------   -------   ---------------
REPLICATED_MTREE           1924.5   RW       ddboost                 -
-----------------   --------------   ------   -------   ---------------
 D    : Deleted
 Q    : Quota Defined
 RO   : Read Only
 RW   : Read Write
 RD   : Replication Destination

Now you can point your Veeam (or whatever) jobs to the new Data Domain with your historical data intact. Yay!

Avamar password recovery

June 04, 2018 by Abe Covello in Avamar, Security

I recently had an issue where, on a new Avamar install, the customer mis-typed the password twice. We had selected the option to make all passwords the same, so we were unable to log into the system at all. We booted the Utility node into Single User Mode by following the instructions here.

After we reset the OS account password we were able to SSH into the Avamar, but we were unable to successfully change all of the passwords. When prompted for the Avamar root account password in the change-passwords script the password we tried didn't work, it was asking us for the incorrectly typed password that set this whole thing in motion.

After SSHing in as admin, we were able to su to root. Then we decrypted the MC Database Tables and this gives us a handful of Avamar account passwords. Because we has chosen to make all passwords the same during the workflow install, we had our mistyped password. Re-encrypt the DB tables and rerun the change-passwords script and everything is ok!

admin@avamar:~/>: su
Password:
root@avamar:/home/admin/#: grep AP /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml
                <entry key="backuprestoreAP" value="{AES}9qRz5orPek4Dq7ybDzh/MA==" />
                <entry key="backuponlyAP" value="{AES}9qRz5orPek4Dq7ybDzh/MA==" />
                <entry key="rootAP" value="{AES}9qRz5orPek4Dq7ybDzh/MA==" />
                <entry key="MCUSERAP" value="{AES}9qRz5orPek4Dq7ybDzh/MA==" />
                <entry key="restoreonlyAP" value="{AES}9qRz5orPek4Dq7ybDzh/MA==" />
                <entry key="viewuserAP" value="{AES}9qRz5orPek4Dq7ybDzh/MA==" />
root@avamar:/home/admin/#: mccipher decrypt --all

**********************************************************************************
* EMC Avamar Management Console (MC).                                            *
* MC Security Tool for Secret Key generation, encryption, decryption and digest. *
**********************************************************************************

WARNING: MCS Preference looks like already decrypted: mcserver:/usr/local/avamar/lib:com/avamar/asn/rmi_ssl_keystore_ap
WARNING: Password is NULL or EMPTY: mcserver:/usr/local/avamar/lib:com/avamar/mc/cac/ldap_login_ap
WARNING: MCS Preference looks like already decrypted: mcserver:/usr/local/avamar/lib:com/avamar/mc/dpn/users/MCUSERAP
WARNING: MCS Preference looks like already decrypted: mcserver:/usr/local/avamar/lib:com/avamar/mc/dpn/users/backuponlyAP
WARNING: MCS Preference looks like already decrypted: mcserver:/usr/local/avamar/lib:com/avamar/mc/dpn/users/backuprestoreAP
WARNING: Password is NULL or EMPTY: mcserver:/usr/local/avamar/lib:com/avamar/mc/dpn/users/mcserver_keystore_ap
WARNING: Password is NULL or EMPTY: mcserver:/usr/local/avamar/lib:com/avamar/mc/dpn/users/mcserver_truststore_ap
WARNING: MCS Preference looks like already decrypted: mcserver:/usr/local/avamar/lib:com/avamar/mc/dpn/users/restoreonlyAP
WARNING: MCS Preference looks like already decrypted: mcserver:/usr/local/avamar/lib:com/avamar/mc/dpn/users/rootAP
WARNING: MCS Preference looks like already decrypted: mcserver:/usr/local/avamar/lib:com/avamar/mc/dpn/users/viewuserAP
WARNING: Password is NULL or EMPTY: mcserver:/usr/local/avamar/var/mc/server_data/prefs:com/avamar/mc/cac/ldap_login_ap
WARNING: Password is NULL or EMPTY: mcserver:/usr/local/avamar/var/mc/server_data/prefs:com/avamar/mc/dpn/users/mcserver_keystore_ap
WARNING: Password is NULL or EMPTY: mcserver:/usr/local/avamar/var/mc/server_data/prefs:com/avamar/mc/dpn/users/mcserver_truststore_ap
WARNING: Can not decrypt some or all MCS Preferences.

All MC DB Tables have been decrypted successfully.

All MCCLI config files have been decrypted successfully.

See MCCipher log for details.

? 2012 EMC Corporation. All rights reserved.
root@avamar:/home/admin/#: grep AP /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml
                <entry key="backuprestoreAP" value="backuprestore1" />
                <entry key="backuponlyAP" value="backuponly1" />
                <entry key="rootAP" value="RootPassword!" />
                <entry key="MCUSERAP" value="MCUser1" />
                <entry key="restoreonlyAP" value="restoreonly1" />
                <entry key="viewuserAP" value="viewuser1" />
root@avamar:/home/admin/#: mccipher encrypt --all

**********************************************************************************
* EMC Avamar Management Console (MC).                                            *
* MC Security Tool for Secret Key generation, encryption, decryption and digest. *
**********************************************************************************

WARNING: Password is NULL or EMPTY: mcserver:/usr/local/avamar/lib:com/avamar/mc/cac/ldap_login_ap
WARNING: Password is NULL or EMPTY: mcserver:/usr/local/avamar/lib:com/avamar/mc/dpn/users/mcserver_keystore_ap
WARNING: Password is NULL or EMPTY: mcserver:/usr/local/avamar/lib:com/avamar/mc/dpn/users/mcserver_truststore_ap
WARNING: Password is NULL or EMPTY: mcserver:/usr/local/avamar/var/mc/server_data/prefs:com/avamar/mc/cac/ldap_login_ap
WARNING: Password is NULL or EMPTY: mcserver:/usr/local/avamar/var/mc/server_data/prefs:com/avamar/mc/dpn/users/mcserver_keystore_ap
WARNING: Password is NULL or EMPTY: mcserver:/usr/local/avamar/var/mc/server_data/prefs:com/avamar/mc/dpn/users/mcserver_truststore_ap
WARNING: Can not encrypt some or all MCS Preferences.

All MC DB Tables have been encrypted successfully.

All MCCLI config files have been encrypted successfully.

See MCCipher log for details.

? 2012 EMC Corporation. All rights reserved.
root@avamar:/home/admin/#: grep AP /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml
                <entry key="backuprestoreAP" value="{AES}9qRz5orPek4Dq7ybDzh/MA==" />
                <entry key="backuponlyAP" value="{AES}9qRz5orPek4Dq7ybDzh/MA==" />
                <entry key="rootAP" value="{AES}9qRz5orPek4Dq7ybDzh/MA==" />
                <entry key="MCUSERAP" value="{AES}9qRz5orPek4Dq7ybDzh/MA==" />
                <entry key="restoreonlyAP" value="{AES}9qRz5orPek4Dq7ybDzh/MA==" />
                <entry key="viewuserAP" value="{AES}9qRz5orPek4Dq7ybDzh/MA==" />
root@avamar:/home/admin/#:

Unable to SSH into an Avamar node after upgrading to 7.5.1?

April 20, 2018 by Abe Covello in Avamar

If you're getting the error "Server unexpectedly closed network connections" after upgrading Avamar to 7.5.1 (from 7.4.1-58 in my case) there is an easy fix!

Download and install the latest version of PuTTY. You're looking for v0.7.

Get it direct here:

https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html

Avamar - MCS service not starting

March 22, 2018 by Abe Covello in Avamar

After powering on an Avamar, the MCS service will not start.

admin@Avamar:~/>: dpnctl status
dpnctl: INFO: gsan status: up
dpnctl: INFO: MCS status: down.
dpnctl: INFO: emt status: up.
dpnctl: INFO: Backup scheduler status: down.
dpnctl: INFO: Maintenance windows scheduler status: enabled.
dpnctl: INFO: Unattended startup status: disabled.
dpnctl: INFO: avinstaller status: up.
dpnctl: INFO: ConnectEMC status: up.
dpnctl: INFO: ddrmaint-service status: up.
dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]

Trying to start the MCS service normally generates an error, and the service will not start.

admin@Avamar:~/>: dpnctl start mcs
dpnctl: INFO: Starting MCS...
dpnctl: INFO: To monitor progress, run in another window: tail -f /tmp/dpnctl-mcs-start-output-26146
dpnctl: WARNING: 1 warning seen in output of "[ -r /etc/profile ] && . /etc/profile ; /usr/local/avamar/bin/mcserver.sh --start"

admin@Avamar:~/>:

admin@Avamar:~/>: dpnctl status
dpnctl: INFO: gsan status: up
dpnctl: INFO: MCS status: down.
dpnctl: INFO: emt status: up.
dpnctl: INFO: Backup scheduler status: down.
dpnctl: INFO: Maintenance windows scheduler status: enabled.
dpnctl: INFO: Unattended startup status: disabled.
dpnctl: INFO: avinstaller status: up.
dpnctl: INFO: ConnectEMC status: up.
dpnctl: INFO: ddrmaint-service status: up.
dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]

Force a MCS restore with:

admin@Avamar:~/>: dpnctl start --force_mcs_restore

  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -
Action: starting all
Have you contacted Avamar Technical Support to ensure that this
  is the right thing to do?

Answering y(es) proceeds with starting all;
          n(o) or q(uit) exits

y(es), n(o), q(uit/exit): y
dpnctl: INFO: gsan is already running.
dpnctl: INFO: Restoring MCS data...
dpnctl: INFO: MCS data restored.
dpnctl: INFO: Starting MCS...
dpnctl: INFO: To monitor progress, run in another window: tail -f /tmp/dpnctl-mcs-start-output-26146
dpnctl: WARNING: 1 warning seen in output of "[ -r /etc/profile ] && . /etc/profile ; /usr/local/avamar/bin/mcserver.sh --start"
dpnctl: INFO: MCS started.
dpnctl: INFO: EM Tomcat is already running, not attempting to restart it
dpnctl: INFO: Resuming backup scheduler...
dpnctl: INFO: Backup scheduler resumed.
dpnctl: INFO: AvInstaller is already running.
admin@Avamar:~/>:
admin@Avamar:~/>: dpnctl status
dpnctl: INFO: gsan status: up
dpnctl: INFO: MCS status: up.
dpnctl: INFO: emt status: up.
dpnctl: INFO: Backup scheduler status: up.
dpnctl: INFO: Maintenance windows scheduler status: enabled.
dpnctl: INFO: Unattended startup status: disabled.
dpnctl: INFO: avinstaller status: up.
dpnctl: INFO: ConnectEMC status: up.
dpnctl: INFO: ddrmaint-service status: up.

Verify the Avamar is now online and good to go.

admin@Avamar:~/>: status.dpn
Wed Mar 21 13:03:48 PDT 2018  [AVAMAR.XIOLOGIX.LOCAL] Wed Mar 21 20:03:48 2018 UTC (Initialized Fri Jun 23 19:03:09 2017 UTC)
Node   IP Address     Version   State   Runlevel  Srvr+Root+User Dis Suspend Load UsedMB Errlen  %Full   Percent Full and Stripe Status by Disk
0.0    10.10.10.100   7.4.1-58  ONLINE fullaccess mhpu+0hpu+0hpu   1 false   0.88 9545  2606806   0.1%   0%(onl:40 )  0%(onl:36 )
Srvr+Root+User Modes = migrate + hfswriteable + persistwriteable + useraccntwriteable

System ID: xxxxxxxxxx@00:oo:00:oo:00:ee

All reported states=(ONLINE), runlevels=(fullaccess), modes=(mhpu+0hpu+0hpu)
System-Status: ok
Access-Status: full

No checkpoint yet
No GC yet
No hfscheck yet

Maintenance windows scheduler capacity profile is active.
  The maintenance window is currently running.
  Currently running task(s): script
  Next backup window start time: Wed Mar 21 20:00:00 2018 PDT
  Next maintenance window start time: Thu Mar 22 08:00:00 2018 PDT

If the Scheduler is not started, restart the scheduler service

Maintenance windows scheduler capacity profile is active.
  WARNING: Scheduler is WAITING TO START until Thu Mar 22 08:00:00 2018 PDT.
  Next backup window start time: Thu Mar 22 20:00:00 2018 PDT
  Next maintenance window start time: Thu Mar 22 08:00:00 2018 PDT
admin@NC-Avamar:~/>: dpnctl stop maint
dpnctl: INFO: Suspending maintenance windows scheduler...
admin@NC-Avamar:~/>: dpnctl start maint
dpnctl: INFO: Resuming maintenance windows scheduler...
dpnctl: INFO: maintenance windows scheduler resumed.

Dell EMC Data Protection patches for Spectre/Meltdown

January 24, 2018 by Abe Covello in Avamar, Data Domain, Security

Dell/EMC has not released any patches for the DPS products yet, but I bet they will be coming soon. After all, Avamar and Data Domain are running on Dell server hardware these days and running Intel processors.

Here is the official product matrix for Dell EMC products, including Avamar, Networker, and some Data Domain products (notably absent is Data Domain physical hardware). The link requires a Dell EMC Support login.

https://emcservice.force.com/CustomersPartners/kA6f1000000FD0gCAG

The important parts are:

Restart Data Domain GUI

September 26, 2017 by Abe Covello in Data Domain

Need to restart the Data Domain GUI? Here's how to restart the services:

Log in to the DD via the CLI as sysadmin, and enter SE privilege mode. The system password is the Data Domain serial number.

# system show serialno

# priv set se

# Enter system password:

Disable http and https services on the Data Domain to bring down the GUI process:

# adminaccess disable http
# adminaccess disable https

Enable http and https services on the Data Domain to re start the GUI process:

# adminaccess enable http
# adminaccess enable https

Wait for a couple of minutes to allow services to fully initialize

This site is now fully SSL encrypted

July 06, 2017 by Abe Covello in General

Yay!

Making free space on a Full Data Domain

July 03, 2017 by Abe Covello in Data Domain, Avamar

Have a Data Domain that's full? Need to free up space so your backups continue to run? Check for old snapshots, expire them, then free up space with a file system cleaning.

Had a customer who's replication Data Domain was significantly (20% more!) more utilized than the primary site, and also was completely full. Which was causing replications to fail. When I checked out the snapshots on one of the Mtrees, I saw that there were 10 snapshots from almost a year ago. These snapshots together were using around 16Tb of the available 67Tb of space on the Data Domain.

SSH into the Data Domain as sysadmin. Use the mtree list command to show the Data Domain Mtree structure.

sysadmin@DataDomain# mtree list
Name Pre-Comp (GiB) Status
---------------------------- -------------- ------
/data/col1/avamar-13896542221487697.5 RW
/data/col1/avamar-1477523534 561707.6 RW
/data/col1/backup 0.0 RW
---------------------------- -------------- ------

Checking the smaller of the Mtrees shows what we expect to see

sysadmin@DataDom1# snapshot list mtree /data/col1/avamar-1477523534
Snapshot Information for MTree: /data/col1/avamar-1477523534
----------------------------------------------
NamePre-Comp (GiB) Create Date Retain UntilStatus
----------------- -------------- ----------------- ----------------- -------
cp.20170622161010 637792.0 Jun 22 2017 09:11 Jun 28 2017 08:50 expired
cp.20170622164953 637792.0 Jun 22 2017 09:51 Jun 28 2017 08:51 expired
cp.20170628151023 656374.0 Jun 28 2017 08:11
cp.20170628155016 661756.7 Jun 28 2017 08:51
----------------- -------------- ----------------- ----------------- -------
 
Snapshot Summary
-------------------
Total:4
Not expired:2
Expired:2

However, when we take a look at the larger of the Mtrees, we see where the problem lies.

sysadmin@DataDom1# snapshot list mtree /data/col1/avamar-1389654222
Snapshot Information for MTree: /data/col1/avamar-1389654222
----------------------------------------------
NamePre-Comp (GiB) Create Date Retain Until Status
----------------- -------------- ----------------- ------------ ------
cp.201609101621301076444.1 Sep 10 2016 09:21
cp.201609111532331100654.4 Sep 11 2016 08:32
cp.201609131705211161823.8 Sep 13 2016 10:05
cp.201609171657211257219.9 Sep 17 2016 09:57
cp.201609181543091284386.1 Sep 18 2016 08:43
cp.201609191705561309786.6 Sep 19 2016 10:05
cp.201609201542051339382.0 Sep 20 2016 08:42
cp.201609211541431359362.4 Sep 21 2016 08:41
cp.201609221542311377744.1 Sep 22 2016 08:42
cp.201609281624221432382.5 Sep 28 2016 09:24
----------------- -------------- ----------------- ------------ ------
 
Snapshot Summary
-------------------
Total: 10
Not expired: 10
Expired:0

Ten snapshots from almost a year ago, taking up so much space is bad. So lets manually expire each snapshot.

sysadmin@DataDom1# snapshot expire cp.20160910162130 mtree /data/col1/avamar-1389654222
Snapshot "cp.20160910162130" for mtree "/data/col1/avamar-1389654222" will be retained until Jul3 2017 08:16.

Do this nine more times to mark all of the old snapshots for deletion.

Checking the snapshots now shows them all expired.

sysadmin@DataDom1# snapshot list mtree /data/col1/avamar-1389654222
Snapshot Information for MTree: /data/col1/avamar-1389654222
----------------------------------------------
NamePre-Comp (GiB) Create Date Retain UntilStatus
----------------- -------------- ----------------- ----------------- -------
cp.201609101621301076444.1 Sep 10 2016 09:21 Jul3 2017 08:16 expired
cp.201609111532331100654.4 Sep 11 2016 08:32 Jul3 2017 08:17 expired
cp.201609131705211161823.8 Sep 13 2016 10:05 Jul3 2017 08:17 expired
cp.201609171657211257219.9 Sep 17 2016 09:57 Jul3 2017 08:17 expired
cp.201609181543091284386.1 Sep 18 2016 08:43 Jul3 2017 08:18 expired
cp.201609191705561309786.6 Sep 19 2016 10:05 Jul3 2017 08:18 expired
cp.201609201542051339382.0 Sep 20 2016 08:42 Jul3 2017 08:18 expired
cp.201609211541431359362.4 Sep 21 2016 08:41 Jul3 2017 08:19 expired
cp.201609221542311377744.1 Sep 22 2016 08:42 Jul3 2017 08:19 expired
cp.201609281624221432382.5 Sep 28 2016 09:24 Jul3 2017 08:19 expired
----------------- -------------- ----------------- ----------------- -------
 
Snapshot Summary
-------------------
Total: 10
Not expired:0
Expired: 10

And now we manually kick off the file system cleaning.

sysadmin@DataDom1# filesys clean start nowait
Cleaning started.Use 'filesys clean watch' to monitor progress.

When the cleaning is finished we should be good to go. In this case we cleared the Avamar logs, kicked off an Avamar checkpoint, and resumed the scheduler.

High density AP environments & the benefit of tunneling

June 22, 2017 by Abe Covello in Wireless

If you're running or designing a centrally controlled WiFi solution involving high density AP placement, such as in a hotel or event center, you can significantly improve AP roaming speed and reduce wireless service interruptions to the end users by using GRE tunnels out to the access points.

I recently had an Wireless network implementation where we had very high AP density in a hotel. 146 APs spread over 5 floors and 265 rooms. We had an issue where a client could be connected to the network, with good signal and low noise, and as they moved around the building, they would sometimes lose internet connectivity for 5 minutes. Their WiFi connection would remain strong but their internet would drop. We messed around with roaming settings, signal strength, anything we could think of. We weren't making any progress. On each floor of the hotel there would be 3 or 4 switches for the APs (and other things) to connect to, giving us 15-20 total switches that all ran back to the network core (through a few intermediate switches). We started tracing ARP and MAC entries on the floor and core switches. We saw that when a client lost internet connectivity while roaming around the hotel that there would be disparate ARP and/or MAC entries on the AP switches, the intermediate switches, and the core. Once the "bad" MAC entry expired, the client would connect to the internet again.

Due to the AP density a roaming client could see anywhere from 3-10 APs at any one time. Those 10 APs might be connected to 10 different switches, connected to 3 different intermediate switches before reaching the core. As the client hopped rapidly between APs, the MAC and ARP refresh couldn't keep up, and so the client lost internet while maintaining excellent wireless network signal.

This particular implementation was running Ruckus H510 APs connected to the SmartZone 100 AP controller. We enabled GRE Tunnels and immediately saw the issue go away. A client could roam freely and never lose internet connection. Looking at the MAC and ARP tables, we saw the client traffic going directly to the SmartZone where it was encapsulated into the GRE tunnel and sent out to the AP.

Avamar - RabbitMQ errors

November 15, 2016 by Abe Covello in Avamar

If you are seeing RabbitMQ errors in your Avamar logs, you'll want to correct the issue. The RabbitMQ error is often due to a mis-configuration of the BRM Service. If you're not using BRM, but the MCS is expecting to see/send data to BRM you can get errors in the logs for the RabbitMQ.

Check your mcserver.xml file for a line "enableBrmService". If set to "true" and you're not using BRM, then it should be set to "false".

admin@AVAMAR:/usr/local/avamar/var/mc/server_data/prefs/>: grep -i enableBRM mcserver.xml

You should see output similar to

<entry key="enableBrmService" value="true" />

If so, edit the mcserver.xml with vi, and change the entry to "false".

Stop, then start the MCS service

mcserver.sh --stop

mcserver.sh --start

Log back into the GUI and clear out the error logs.

You should not see the RabbitMQ errors any more.

Avamar Proxy username and password

August 31, 2016 by Abe Covello in Avamar

I don't always log on to an Avamar proxy, but when I do, I never remember the default logon info.

Incidentally, it's:

root

avam@r

Avamar: Root to root Migration

August 15, 2016 by Abe Covello in Avamar

Moving from one Avamar to a new Avamar for whatever reason isn't particularly hard, but can present some challenges, especially since EMC support kicks you off to EMC Professional Services. There are guides floating around, and this is all based off of those guides.

First thing to do is create/add an exception to the GSAN port file. This will allow the unencrypted or encrypted traffic necessary to perform the root-to-root migration. Even if you can telnet on ports 27000 and 29000 from all nodes to all other nodes, you'll need to add this in.

On both the source (old) node(s) and the destination (new) node(s), edit the gsan-port file (or create it if it isn't there). Logon to the Utility node as admin, then -su to root. Load the dpnid keys with

ssh-agent bash
ssh-add ~admin/.ssh/admin_key

Create (or edit) the gsan-port file using VI

vi /usr/local/avamar/lib/admin/security/gsan-port

On the first line of the file add in the port exception (27000 for unencrypted and 29000 for encrypted)

GSAN_PLAIN_TEXT='27000,'

Save the file and exit VI.

If you're on a grid, copy the file to all nodes with

mapall --user=root copy /usr/local/avamar/lib/admin/security/gsan-port

Then, copy the file to the correct directory with

mapall --user=root mv /root/usr/local/avamar/lib/admin/security/gsan-port /usr/local/avamar/lib/admin/security/

Restart the Avamar firewall services

mapall --noerror --all+ --user=root 'service avfirewall restart'

If you're using a Data Domain as your backup data target, add the Data Domain to the destination Avamar.

Now we can get started with the actual migration.

On the source server -su back to admin and flush the MCServer

mcserver.sh --flush

On the destination server, stop the MCS service with

dpnctl stop mcs

On the source server, run the migration command, where DST is the destination Avamar IP address or hostname, DST-ROOT-PASSWORD is the destination root password, and SRC-ROOT-PASSWORD is the source root password.

Note, there are two ROOT users in Avamar, "Avamar Root" and "OS Root". This is the "Avamar Root" user credentials we're looking for here. You can verify the credentials by running the following command from both the source and destination Avamar nodes.

avmgr logn --id=<username> --ap=<password> --debug

Migration Command:

nohup avrepl --operation=replicate --[replscript]dstaddr=DST --[replscript]dstid=root --dstpassword=DST-ROOT-PASSWORD --[avtar]id=root --[replscript]fullcopy=true --ap=SRC-ROOT-PASSWORD --send-adhoc-request=false --max-streams=8 --[replscript]timeout=0

You can watch the progress of the replication by SSHing back into the source Avamar and running

tail -f /home/admin/nohup.out

Root-to-root migration can take anywhere from several hours to several days, depending on the amount of data to transfer.

Prior to cutover, you should run the migration command again to catch any new data that was put onto the source server after the replication command was first run.

When you're ready to cutover, SSH into the destination Avamar as admin and run

ssh-agent bash
ssh-add ~admin/.ssh/admin_key

To load the SSH key, then

mcserver.sh --restore --restoretype=new-system

Follow the prompts as requested by Avamar to complete the migration.

Start the MCS service on the destination server with

dpnctl start mcs

Take a checkpoint on the new destination Avamar and verify data with a test restore or two.

UPDATE - Apparently, if you're migrating from a version previous to 7.3 up to 7.3, you need an undocumented switch in your migration command. Add

--[replscript]dstencrypt=proprietary

to your migration command and you'll be off and running.

Your migration command will look something like:

nohup avrepl --operation=replicate --[replscript]dstaddr=<destination FQDN> --[replscript]dstid=root --dstpassword=<destination password> --[avtar]id=root --[replscript]fullcopy=true --ap=<source password>--[replscript]dstencrypt=proprietary --send-adhoc-request=false --max-streams=8 --[replscript]timeout=0 --debug &

Avamar: single node sizing

August 09, 2016 by Abe Covello in Avamar

When you replace a grid with a single node, there is often concern from customers about the amount of metadata on the grid, and if a single node will have enough storage. Regardless of the size of the grid, the metadata space is limited to 3.9TB, which (depending on the amount of deduplication you're getting) amounts to somewhere in the 500-750TB of backup data range. Conveniently, an M1200 utility node has 3.9TB of useable space. So, regardless of the size of your grid, it can be replaced with a M1200 Single Node Avamar. As long as you don't store any data on the node itself. There is a procedure to lock the Avamar from receiving backup data, and it's a good idea to run the procedure as part of the migration (or new installation) of a single node Avamar.

Data Domain - reset sysadmin password

Xiologix

March 10, 2016 by Abe Covello in Data Domain

I believe that the shell-escape command will no longer work as of DDOS 6.1.x without a encrypted password from Dell/EMC that is changed often (daily?). Sorry folks. :(

Every now and then I run into a situation where a customer has forgotten the sysadmin password to their Data Domain. You can't reset the sysadmin password from the CLI or GUI unless you are logged into the sysadmin account, here's how to do it from bash (it is after all just a Linux box), as long as you have another account on the system with admin access.

Note: You will be putting your Data Domain in a situation where you can, with the wrong command, completely destroy your data. If you do so, I'm not responsible, EMC isn't responsible, you're responsible. So be responsible, and be careful. Also, this is all covered in the Data Domain documentation. So I'm not sharing some huge secret here. Mostly just archiving it so I have a convenient place to go back to the next time I have to do this (which is probably in another month or so). Finally, if you don't have another account on the Data Domain apart from the sysadmin, this won't work (as it requires SSHing into the Data Domain) and you'll have to call support.

That being said, here we go.

First, SSH into the Data Domain as a user with admin credentials. Then:

system show serialno
priv set se
(Password is SN)
ctrl-c
ctrl-c
ctrl-c
uname
fi st
df
ctrl-c
ctrl-c
ctrl-c
shell-escape
passwd sysadmin
type in your new password twice, then
exit

That's all there is to it. Have fun, and try not to break anything.