Difference between revisions of "Zippy"

From CHG-Wiki
Jump to navigationJump to search
m
m (→‎Notes: RMA sdh note.)
 
(13 intermediate revisions by the same user not shown)
Line 1: Line 1:
−
Zippy has been CHG's workhorse server, but is currently being rebuilt, with functionality being shifted over to [[rain]] due to zippy's ailing hardware and configuration.
+
Zippy is CHG's redone SunFire X4600 workhorse server. Most functionality of the old Zippy has been shifted over to other systems. Zippy now mainly functions as scratch space (raida) and backup space (raidb). Zippy also houses small experimental/personal VMs.
 +
 
 +
<!--Depreciated after Fedora24 rebuild '''Please note zippy's [[Zippy#Reboot_Sequence|reboot sequence]] below.''' -->
  
 
==General Info==
 
==General Info==
 
{| class="wikitable"
 
{| class="wikitable"
 
! System Name:  
 
! System Name:  
−
|align="right"|zippy.geog.ucsb.edu
+
|align="right"|zippy.chg.ucsb.edu
 
|-
 
|-
 
! IP:  
 
! IP:  
−
|align="right"|128.111.234.240
+
|align="right"|128.111.236.130 <!--used to be 234.240-->
 
|-
 
|-
 
! IPMI IP:  
 
! IPMI IP:  
−
|align="right"|128.111.234.239
+
|align="right"|<em>(need info)</em> <!--used to be 128.111.234.239-->
 
|-
 
|-
 
! Location:  
 
! Location:  
−
|align="right"|EH 1609
+
|align="right"|EH ERI Server Room
 
|-
 
|-
 
! UCID #:  
 
! UCID #:  
Line 45: Line 47:
 
==System Configuration==
 
==System Configuration==
 
*OS Type: Unix
 
*OS Type: Unix
−
*OS Version: Red Hat Linux (expired license)
+
*OS Version: FedoraCore24 <!-- used to be Red Hat Linux (expired license), the CentOS 6.7 -->
 
*CPU info: 32xAMD 2.86GHz (may only be 16 with two cores)
 
*CPU info: 32xAMD 2.86GHz (may only be 16 with two cores)
−
*Chassis Specs: Takes up lots of rack space, three different RAID chunks (gibber, jabber, jower) <em>(more info needed)</em>
+
*Chassis Specs: SunFire X4600
 +
**Defunct RAID chunks (gibber, jabber, jower [HD: 25TB, RAID6])
  
 
===Network===
 
===Network===
Line 57: Line 60:
 
*Memory: 160GB (DDR2?)
 
*Memory: 160GB (DDR2?)
 
*HD: 25TB, RAID6
 
*HD: 25TB, RAID6
 +
 +
<!--Depreciated after Fedora24 rebuild  ===Reboot Sequence===
 +
Zippy needs a very specific order to its reboot process.
 +
#Shut down zippy with <code>shutdown -h now</code>
 +
#Manually (physically) shut down external RAID arrays (they will not shut down with zippy).
 +
#Manually power zippy on, wait for system to completely load.
 +
#Once zippy's system is completely up, power on external RAID arrays.
 +
#Remount RAID arrays.
 +
-->
  
 
==Services==
 
==Services==
−
Recently rebuild, TDB. Formerly Lis, CSCD1 (most tasks shifted to rain).
+
As of the FedoraCore24/ZFS rebuild, zippy is mainly a scratch/backup space (including BackupPC) that also hosts personal/experimental VMs.  
 +
 
 +
Formerly Lis, CSCD1 (most tasks shifted to rain or ubu).
 +
 
 +
==Notes on October 2013 Redo==
 +
===Troubleshooting===
 +
Timed zippy's startup at two and a half minutes (booted much faster when we removed some of the RAM/CPUs). Had some graphical issues - screen would go black before kickstarter could really initialize. The system seemed to lock up when trying to access disks or the SAS configuration utility with a live (but blank) screen. After swapping around drives and taking out various cards, switched to a standalone monitor as opposed to KVM, which resolved the issue.
  
−
===Notes===
+
=== Day 1 ===
−
Old Zippy RAIDS, now locked or defunct(?) - gibber, jabber, jower;
+
First attempt (specifying a build on sda,sdb) came up with error about not enough disk space, so tried (sdb,sdc) but that failed with no device found. Added the --all to the initpart in kickstart and then it hung at the storage device initialization. Left for 45 minutes with blank "live" screen, but it was still unbootable afterwards. Subsequent attempts did the same.
 +
 
 +
We tried a number of variations and had one pass where CPUs 8,9,10,11,12,14 were hung. The boot process on the following pass seemed much faster for some reason. Had a few other passes where there were other odd little inconsistencies.
 +
 
 +
We tried a subsequent pass with all but two of the CPU cards pulled and another with the fiber channel controllers pulled, with no change in results.
 +
 
 +
=== Day 2 ===
 +
Attempt to access the LSI card BIOS, but got same black screen hang when the config utility started (like the one we got when the jumpstart install got to the disk config).
 +
 
 +
Tried disabling PXE boot stuff, but after a full power cycle, it seemed like it started attempting the PXE boots again.
 +
 
 +
===ILOM Info===
 +
====Serial Connection====
 +
Connect using 9600,N,8,1
 +
cd /SP/users/root
 +
show
 +
set password=newpass
 +
====Networking====
 +
Set the pending network settings (you can't set the regular ones)
 +
cd /SP/network
 +
set pendingipaddress=128.111.101.114
 +
...
 +
 
 +
Now commit the changes
 +
set commitpending=true
 +
 
 +
Check that settings updated
 +
show
 +
 +
Reboot the Service Processor (equivalent to a full power up (i.e.: plugging in initially))
 +
reset /SP
 +
 
 +
To stop/start the system
 +
stop /SYS
 +
start /SYS
 +
 
 +
To get at console (NOTE: Had issues escaping the console)
 +
start /SP/console
 +
 
 +
Go in via SSH and kill the console
 +
stop /SP/console
 +
 
 +
===Network Management Port===
 +
Once configured:
 +
ssh root@address
 +
 
 +
then things are just like the serial console...
 +
 
 +
===Links===
 +
*ILOM help: [http://docs.oracle.com/cd/E19350-01/E21356/6_config-booting.html#50634778_89866]
 +
*Notes ganked from [http://wiki.eri.ucsb.edu/sysadm/Zippy].
 +
 
 +
==Notes on August 2016 Redo==
 +
Coming soon!
  
 
==VMs==
 
==VMs==
Line 70: Line 141:
 
! Notes:  
 
! Notes:  
 
|-
 
|-
−
|chg-helmet
+
|[[helmet]]
−
|128.111.234.246
+
|128.111.236.136 <!--used to be 234.246-->
−
|Shrad's sandbox VM
+
|Moved to [[ubu]].
 +
|-
 +
|[[fez]]
 +
|N/A <!--used to be 128.111.234.169-->
 +
|No longer in use - nuked.
 +
|-
 +
|[[chg-ewx]]
 +
|no longer on zippy, back on [[ubu]]
 +
|EWX host, temporarily lived on zippy
 
|}
 
|}
 +
 +
==Notes==
 +
*Zippy was moved from EH 1609 to the sixth floor EH ERI server room on November 23rd, 2015. Its domain (and the domain on its VMs) were updated from geog.ucsb.edu to chg.ucsb.edu at that time.
 +
*RMA'd [?] drive to HGST (Hitachi) on July 7th, 2014 (used ubu shelf spare to replace).
 +
*RAM errors have been a known issue for a while. According to [[mtc]], "It's ECC so they're only "notices" and the system deals safely w/ the issue. It's technically feasible to track them down, probably a good idea to do so, but can be difficult to do, especially give the number of sticks in zippy." [[User:Libby|Libby]] 13:07, 3 August 2016 (PDT)
 +
*Aaron turned off CPU 10 on October 23rd, 2016 (approximately), which bought us a little more time with the system (the CPU in question was causing errors which originally indicated an end-of-life state for zippy).
 +
*In October of 2016, zippy's two ZFS RAIDs (/raida and /raidb) were combined into a single RAID (/raid).
 +
*After the revamping of zippy with FedoraCore24 and ZFS, the package "libXp-1.0.2-6.fc24.x86_64" (dnf install libXp-1.0.2-6.fc24.x86_64) was needed to get ENVI/IDL working. This package is generally required for fresh installs. [[User:Libby|Libby]] 09:37, 2 November 2016 (PDT)
 +
*RMA'd sdh drive to Western Digital on November 16th, 2016 (failed November 7th).
  
  
 
[[Category:Hosts]]
 
[[Category:Hosts]]

Latest revision as of 16:10, 16 November 2016

Zippy is CHG's redone SunFire X4600 workhorse server. Most functionality of the old Zippy has been shifted over to other systems. Zippy now mainly functions as scratch space (raida) and backup space (raidb). Zippy also houses small experimental/personal VMs.


General Info

System Name: zippy.chg.ucsb.edu
IP: 128.111.236.130
IPMI IP: (need info)
Location: EH ERI Server Room
UCID #: 088000240
Grant #: (need info)
Serial #: (need info)
General Purpose: Workhorse server
Purchase Date: (need info)
Delivery Date: (need info)
Vendor: (need info)
Contract #: (need info)
Support Expires: (need info)

System Configuration

  • OS Type: Unix
  • OS Version: FedoraCore24
  • CPU info: 32xAMD 2.86GHz (may only be 16 with two cores)
  • Chassis Specs: SunFire X4600
    • Defunct RAID chunks (gibber, jabber, jower [HD: 25TB, RAID6])

Network

  • NIC speed: Gigabit
  • MAC Address - eth0: 00:14:4F:D1:E9:10
  • MAC Address - eth1: (need info)

Storage

  • Memory: 160GB (DDR2?)
  • HD: 25TB, RAID6


Services

As of the FedoraCore24/ZFS rebuild, zippy is mainly a scratch/backup space (including BackupPC) that also hosts personal/experimental VMs.

Formerly Lis, CSCD1 (most tasks shifted to rain or ubu).

Notes on October 2013 Redo

Troubleshooting

Timed zippy's startup at two and a half minutes (booted much faster when we removed some of the RAM/CPUs). Had some graphical issues - screen would go black before kickstarter could really initialize. The system seemed to lock up when trying to access disks or the SAS configuration utility with a live (but blank) screen. After swapping around drives and taking out various cards, switched to a standalone monitor as opposed to KVM, which resolved the issue.

Day 1

First attempt (specifying a build on sda,sdb) came up with error about not enough disk space, so tried (sdb,sdc) but that failed with no device found. Added the --all to the initpart in kickstart and then it hung at the storage device initialization. Left for 45 minutes with blank "live" screen, but it was still unbootable afterwards. Subsequent attempts did the same.

We tried a number of variations and had one pass where CPUs 8,9,10,11,12,14 were hung. The boot process on the following pass seemed much faster for some reason. Had a few other passes where there were other odd little inconsistencies.

We tried a subsequent pass with all but two of the CPU cards pulled and another with the fiber channel controllers pulled, with no change in results.

Day 2

Attempt to access the LSI card BIOS, but got same black screen hang when the config utility started (like the one we got when the jumpstart install got to the disk config).

Tried disabling PXE boot stuff, but after a full power cycle, it seemed like it started attempting the PXE boots again.

ILOM Info

Serial Connection

Connect using 9600,N,8,1 

cd /SP/users/root
show
set password=newpass

Networking

Set the pending network settings (you can't set the regular ones) 

cd /SP/network
set pendingipaddress=128.111.101.114
...

Now commit the changes 

set commitpending=true

Check that settings updated 

show

Reboot the Service Processor (equivalent to a full power up (i.e.: plugging in initially)) 

reset /SP

To stop/start the system 

stop /SYS
start /SYS

To get at console (NOTE: Had issues escaping the console) 

start /SP/console

Go in via SSH and kill the console 

stop /SP/console

Network Management Port

Once configured: 

ssh root@address

then things are just like the serial console...

Links

  • ILOM help: [1]
  • Notes ganked from [2].

Notes on August 2016 Redo

Coming soon!

VMs

System Name: IP: Notes:
helmet 128.111.236.136 Moved to ubu.
fez N/A No longer in use - nuked.
chg-ewx no longer on zippy, back on ubu EWX host, temporarily lived on zippy

Notes

  • Zippy was moved from EH 1609 to the sixth floor EH ERI server room on November 23rd, 2015. Its domain (and the domain on its VMs) were updated from geog.ucsb.edu to chg.ucsb.edu at that time.
  • RMA'd [?] drive to HGST (Hitachi) on July 7th, 2014 (used ubu shelf spare to replace).
  • RAM errors have been a known issue for a while. According to mtc, "It's ECC so they're only "notices" and the system deals safely w/ the issue. It's technically feasible to track them down, probably a good idea to do so, but can be difficult to do, especially give the number of sticks in zippy." Libby 13:07, 3 August 2016 (PDT)
  • Aaron turned off CPU 10 on October 23rd, 2016 (approximately), which bought us a little more time with the system (the CPU in question was causing errors which originally indicated an end-of-life state for zippy).
  • In October of 2016, zippy's two ZFS RAIDs (/raida and /raidb) were combined into a single RAID (/raid).
  • After the revamping of zippy with FedoraCore24 and ZFS, the package "libXp-1.0.2-6.fc24.x86_64" (dnf install libXp-1.0.2-6.fc24.x86_64) was needed to get ENVI/IDL working. This package is generally required for fresh installs. Libby 09:37, 2 November 2016 (PDT)
  • RMA'd sdh drive to Western Digital on November 16th, 2016 (failed November 7th).
Retrieved from "https://wiki.chc.ucsb.edu/index.php?title=Zippy&oldid=108673"