Pio's work related musings

Pio's work related musings

David  //  

Apr 7 / 1:11am

Unknown status for servers in ITAssistant

ITAssistant often seems to give unknown status to servers for no apparent reason.  The usual fix for this is just to refresh the inventory of the unknown servers (annoying but quite quick to do).

However sometimes servers go unknown and won't respond to an inventory, even if you bounce the ITAssistant services or the SNMP services on the remote server.

The first step is to try the troubleshooting tool in ITAssistant...

Ita-unknown-1

and run the SNMP connectivity test...

Ita-unknown-2

The result should show a list of SNMP agents installed on the server.  A healthy Dell Openmanage report will list the OpenManage Server Agent, plus any additional components that were selected (such as storage management.

If the OpenManage SNMP agent isn't showing you'll probably only see a couple of agents.

e.g.
1) A problem server
Connected to the agent software(s) - [broadcom, NA], [mib2, NA]
2) An OK server
Connected to the agent software(s) - [broadcom, NA], [cminventorysnmp, NA], [drac3, NA], [mib2, NA], [OpenManage Server Agent, 5.8.0], [storagemgmt, NA]

If the server responds with an output similar to the problem server (1) above, then you need to reinstall SNMP.  The good news is that this can be done without rebooting the server.

This usually requires the unzipped files for the installed service pack and the windows installation CD, so make sure you have the media available

1) Uninstall Dell OpenManage
2) Uninstall SNMP
3) Reinstall SNMP
4) Reinstall Dell OpenManage

Finally go back into ITAssistant and re-run the troubleshooting tool to make sure you get the full SNMP output for OpenManage and if this is ok, re-inventory the server in OpenManage and your unknown status should clear.

Of course if you take the opportunity to upgrade OpenManage you may find your firmware is out of date and you get a warning status, but that's a different subject...

Filed under  //  dell   itassistant  
Feb 12 / 6:53am

Dell iSCSI Tape Library devices not appearing

The Dell TL2000 and TL4000 tape libraries both have the option to be used with iSCSI.  This comes in the form of a SAS to iSCSI bridge which installs in the unit.  The SAS ports are then connected to the Bridge card with an adapter cable (supplied) and the network ports on the card  are connected to your iSCSI network.

The microsoft iSCSI initiator is used to connect to the iSCSI bridge which presents the tape library hardware as a series of targets.

At first glance the first target appears to be duplicated but do not be fooled (like I was).  These are actually the first tape drive *and* the robotic library itself.

Once all the devices are logged on, you should see the Robotic library and all the drives in your changer in backup exec although you may need to rerun the device detection wizard in BE first.


(download)

Filed under  //  TL2000   TL4000   dell  
Oct 21 / 4:26am

Monitoring VMware vSphere ESX4 with Dell ITAssistant with SNMP

This is a howto for setting up SNMP on VMware vSphere ESX4 to work with ITAssistant, written because there's precious little documentation on the matter and what documentation exists is quite often misleading or plain wrong!  Vmware in their wisdom have completely changed the way SNMP is configured in vSphere ESX4 (note not ESX4i which is a completely different kettle of fish and covered elsewhere).  This means (young padawan) you have to unlearn everything you knew and any documentation for pre-vSphere can be pretty much thrown out of the window.  The version used was 4.0.0 but should apply to future versions too.

Install Openmanage 6.1.x or higher

This is done the same way as on older vmware servers or linux systems.

1) download and unzip the Openmanage redhat package (OM_6.1.0_ManNode_A00.tar.gz) from the dell website to someone on in your vmware service console.

gzip -d OM_6.1.0_ManNode_A00.tar.gz
tar -xvf OM_6.1.0_ManNode_A00.tar

This will give the following files/volders:
COPYRIGHT.txt
docs/
license.txt
linux/
setup.sh

2) Run setup (as root) and select the first 3 options
sh setup.sh

##############################################

Server Administrator Custom Install Utility

##############################################

 

  Components for Server Administrator Managed Node Software:

 

    [x] 1. Server Administrator Web Server

    [x] 2. Server Instrumentation

    [x] 3. Storage Management

    [ ] 4. Remote Access Core Components

    [ ] 5. Remote Access SA Plugin Components

    [ ] 6. All

 

  Enter the number to select a component from the above list.

  Enter q to quit.

 

Then select i to install

3) start the Openmanage services
srvadmin-services.sh  start

4) Open the vmware firewall to allo access to the openmanage web interface

/usr/sbin/esxcfg-firewall -o 1311,tcp,in,OpenManage

Make sure you test it so you know openmanage is working and can see your hardware.  If you can see the disk systems, memory, cpus etc, then fix openmanage first.

Setup passive polling from ITAssistant

Passive monitoring is where the ITAssistant polls the server at a regular interval.  Supported methods in ITAssistant are CIM, SNMP and IPMI.  SNMP with the "get" string is the usual method and what this howto covers.  The ITAssistant documentation recommends creating a new readonly string for use with ITAssistant.

1) Edit the snmp conf file
/etc/snmp/snmpd.conf

2) Locate line 41 which should contain this string:
com2sec notConfigUser  default       public

3) Change the value from public to your string used with discovery in ITAssistant.

4) Append the following to the bottom of the file
smuxpeer .1.3.6.1.4.1.674.10892.1

This allows the snmp service to also serve out the Dell MIB information.

5) Restart the SNMPD service
/sbin/service snmpd restart
It often shows "failed" for the stop part but you can ignore that.

6) Restart the Dell Openmanage services for good measure
srvadmin-services.sh restart

7) Open up the SNMP service on the vmware firewall:
/usr/sbin/esxcfg-firewall -e snmpd

7) Test SNMP with the troubleshooting tool in ITAssistant (tools->troubleshooting tool).  Select the SNMP connectivity test and click the configure button to change the get string to your string (if you're not using

Alternatively use snmpwalk on the Dell MIB Tree
snmpwalk -v2c -c <your-string> 127.0.0.1 .1.3.6.1.4.1.674.10892.1
This should produce alot of scrolling text as it prints all the snmp output.

I'm still ironing out the quirks with active alerts (SNMP traps) and hope to have a definitive guide soon!

Filed under  //  dell   itassistant   vmware  
Sep 29 / 11:33pm

Wake on LAN problems - Dell 2950 servers and Broadcom NICS

Although I've been using vmware for over 3 years through various incarnations, I've only just been in a position to exploit the power-saving features in v3.5. 

At first glance it looks pretty straight forward - click the "enter standby" option and sit back and watch. No.

First off, you need to have Wake on LAN (WOL) supported on your vmotion NIC.   This isn't necessarily something you think of when setting up the patching of the virtual server - Intel PT1000 Quad Port NICs only support WOL on the first port (Dual ports also).  Others in the range might be different, but the VT1000 NICs have their own issues in vmware depending on the version of vmware installed.

Most of the servers I work on are Dell x9xx series which have Broadcom 5708 onboard NICs.  Looking in Virtual Center at the NIC configuration shows whether VMware thinks WOL is available.  I was a bit confused when I found for some servers it was available and for others it wasn't.  Dell support went through the usual steps - update the firmware, update the NIC firmware, etc. - to no avail.

I tested the WOL feature anyway with the AMD Magic Packet utility and the MAC address of the NIC and lo! it woke up.  Curiouser and curiouser.

After more research by Dell, one of their engineers discovered source of the issue (hurrah!) in the version of the Ethernet Controller Hardware.  an lspci command (run as root) reveals the version number:

05:00.0 Ethernet controller: Broadcom Corporation Broadcom NetXtreme II BCM5708 1000Base-T (rev 11)

Rev 11 is the culprit and Rev 12 servers correctly show WOL available in vmware.  Dell have escalated the issue to vmware so hopefully there will be some kind of resolution soon.

So in summary:

  1. Make sure vmotion is enabled on a NIC that supports WOL
  2. Check your broadcom NICs are Rev12 and above.
  3. Put your Host in a Cluster
  4. At least one other Host in the Cluster must be on (this does the wakeup on the others that are in standby)
  5. Enable WOL in the NIC boot bios

Blogged with the Flock Browser
Filed under  //  2950   broadcom   dell