Open Systems SnapVault (OSSV) Web Reporting

Open Systems SnapVault (OSSV) is a data replication/protection software from NetApp. OSSV runs on Windows, UNIX/LINUX, and VMWare ESX servers and it replicates block level changes back to a NetApp filer. We use this technology to replicate the data on all of branch office servers back to our datacenter. This replication allowed us to remove the need to perform tape backups in each branch office.

One of the strengths of OSSV is its simplicity, but that is also it’s weakness. The reporting and monitoring capabilities of OSSV can be cumbersome. You can get a lot of status information from the filer itself via the snapvault status command. However, the one thing that you can’t tell from filer is how far along the process is. That information is only available when the snapvault status -l command is run on the primary system (your Windows/UNIX/VMWare server). With windows that means that you either need to get either console access (via. Remote Desktop, VNC, etc.) or command line access with something like PsExec.

We’ve been running this way for almost two years, but I finally got tired of all of the typing. So I developed a web based reporting tool that gathers all of information and presents it in a nice graphical clickable interface.

The front screen of this php web application displays the status of all currently configured snapvault relationships. It does this by executing the snapvault status via an encrypted ssh key-based session to the filer. The resulting output is shown below.

OSSV Web Reporting Overview Screen

For each specific qtree you can click to get the detail about that status of that relationship just as if you had run the snapvault status -l command on the primary server. Thereby showing you how many files have been transferred, how far along you are on the current file and what errors have been encountered. All this is accomplished via winexe linux command. Winexe remotely executes commands on WindowsNT/2000/XP/2003 systems from GNU/Linux just like PsExec does from windows.

OSSV Server Detail Web Report

If anyone is interested in using this, I can supply you with my really ugly code. Finally a big shout out the phpSysInfo project from where I borrowed the CSS styling.

Update: I’ve finally put the code online for anyone to access.  You can download the original code from my box.net account. Get the ossvreport.zip file.

Update 2: Box.net is no longer allowing easy public access so I’ve put it up on dropbox.com and you should be able to get the ossvreport.zip file from there now.

Virtualization Gripes (part 1)

Virtualization is not new, it has been around for a number of years. So why are so many software vendors afraid of it, and even worse hostile to it? We run a number of different application that have license managers for their software. These license managers are often a service and sometimes a hardware lock / dongle that runs on a server that all of the clients connect to for permission to use the software.

We’ve successfully virtualized many of these servers (they are great candidates because they are so often extremely under utilized when on physical boxes). The ones with hardware dongles pose some problems (I’ll save that for another post). Then along comes a product from ETAP. They will let you virtualize their application (which is extremely resource intensive), but they prohibit (i.e. software won’t run) you from virtualizing their license manager. This is completely insane! Their silly little license manager use very little memory, and almost zero CPU cycles. In our case it may handle one or two requests for a license a day (if that), and I have to install it on a physical box. What a PIA. 

In a plea to all software developers, “you have nothing to fear from allowing your software to be virtualized.” We are not trying to steal your software, we’re just trying to maximize our investment. Let alone allow us to be better prepared for disaster recovery. In fact, by making it difficult/impossible to virtualize is going to cost you money in the long run as I will strongly denounce your product and actively push our company (and others I deal with) to avoid your products and find alternatives that support virutalization.

Virtualization Begins

For about the past nine months I’ve been working on growing the use of virtualiztion within my firm. We had dabbled with virtualizing a couple of misc. application/development servers with the first release of VMWare Server, but I knew that in order to consolidate the rest of environment, as well as better prepare us for disaster recovery that we needed to expand our server virtualization strategy. The solution that made the most sense at the time was a VMWare Infrastructure solution utilizing their ESX platform.

The question was how to move forward without spending tons of money (and avoiding all of the politics involved in deploying a solution). The solution arose through the timing on our computer leases. We lease the majority of our computer equipment for three years. It just so happened that as I was looking to move forward, one of our major leases came up for replacement. To make it even better, not only did I have three servers that were due for swapping (and great candidates for virtualizing), but three years earlier we were forced in to ordering a large batch of workstation class computers. This time, with the advancement of technology, we no longer needed that class of machine.

With my good fortune, I was able to replace the three servers with two new servers (albeit much beefier units) along with the full ESX suite for both, and workstations with much better (thanks to Intel’s Core2Duo chips) desktop class units for basically the same monthly payment we already spending. It was a really win-win for company.

In some upcoming posts I plan to highlight our journey, cover some of the sites that helped us to get where we are, discuss where we are going, and finally highlight some of the difficulties and frustrations that we still face.

Sparklining Excel

According to Wikipedia a sparkline is a

“small, high resolution graphics embedded in a context of words, numbers, images … Whereas the typical chart is designed to show as much data as possible, and is set off from the flow of text, sparklines are intended to be succinct, memorable, and located where they are discussed. Their use inline usually means that they are about the same height as the surrounding text.”

A group has released an open source add-in for Microsoft Excel called TinyGraphs “that creates sparklines, mini column graphs, and area graphs from a row of data. It generates beautiful tiny graphics that are as small as a cell and is useful for visualizing large quantities of data, such as stock prices and exchange rates.”

While I have yet to actually use the add-in it is something that I definitely can see using in a number or reports that I generate at the office.

Review: SB EventLog Monitor

SB EventLog MonitorI have only one thing to say about this product, “How did I ever live without it”. If you manage more than one Microsoft Windows Server then you definitely need to be using SB EventLog Monitor.

So what does SB EventLog Monitor do that is so great, it collects, collates, and reports via a web interface upon Microsoft Event Log data. The UNIX world has had syslog forever and a ton of tools to help you manage the logging data generated by servers. I’ve even tried to shoehorn Microsoft Event Log data into some of those products, but it was never a good fit. SB EventLog Monitor allows you to quickly and easily manage and analysis what is going on across all of your servers. It allows you to quickly and easily view and filter error messages from different servers and identify patterns. This is particularly useful with dealing with multiple servers across slow WAN links.

It collects the Event Log data either via a Microsoft VB script that use WMI to collect only the new events or via an agent that you can install on your servers. The other requirements are MySQL, PHP (5.0+), and a web server (apache, IIS). While the install is geared towards running everything on a Microsoft server it is possible to run the database and web server on Linux. In fact that is what I did. The install is really pretty easy, so if you are looking for a relatively simple way to increase the manageability of your servers, then I strongly recommend that you take the time to install the open source SB EventLog Monitor.

Review: Script Your Documentation Instantly

If you are anything like me you probably have little to no documentation on your servers. Probably it is because you don’t have the time or the personal to perform the tedious (and boring) work required; you know it is important, but other things seem to take priority. Well, you no longer have any excuse for not getting it done.

SYDI (Script Your Documentation Instantly) is an open source solution that will document your Windows Servers, MS SQL Servers, and Exchange Organizations. It is a fantastically easy product to use. In its simplest form, it will query an individual server and produce a Microsoft Word document detailing the hardware, software, networking, user accounts and storage settings with a table of contents and loads of other useful information. With a little extra work, you can have it query all of your servers and produce a set of XML files that can be converted in to HTML documents (using an included script) that makes publishing a breeze.

Using SYDI is really simple; it is just a VBS script that is launched from a command prompt. You do need to have Microsoft Word installed on the workstation if you want it to produce the documentation in that format. The software is written by an IT Consultant named Patrick Ogenstad. He has posted some really good how-to guides on his site.

I highly recommend that you take a look at SYDI and use it to help jump start you server documentation project. I did and in about 30 minutes I had published our server configs on our Network Management server.

New Software to Evaluate

While browsing through the depths of Sourceforge I ran across a number of different software packages that really caught my attention. I will begin looking at them in more depth over the next couple of days. I hope to write some reviews of the packages. If you’ve used any of the following software packages I’d love to hear you experience, or if there are ones that you think I should look at then please leave me a comment.

  • SYDI – Network Documentation Project – SYDI is a project aimed to help system administrators in getting started with their Network Documentation. It can document Windows Servers, MS SQL Servers, Exchange Organizations and Linux systems.
  • phpMSAdmin is a tool written in PHP that allows you to administer a Microsoft SQL Server through a web browser, without the need for Windows or the proprietary Enterprise Manager. It allows you to create/modify: databases, tables, views, triggers, etc.
  • SSLBridge An AJAX enabled Samba web GUI that provides users access to corporate documents without any specialized hardware or software clients.
  • RackMonkey is a web-based tool for managing racks of equipment such as web servers, video encoders, routers and storage devices. Using a simple interface you can keep track of what’s where, which OS it runs, when it was installed, who it belongs to etc. RackMonkey is open source software licensed under the GPL.
  • Davenport is a servlet-based WebDAV gateway to a CIFS network. This allows you to access Windows/Samba shares using any web browser. WebDAV clients (such as Windows Web Folders) can upload and download from the shares as if they were local folders.
  • Clonezilla is a partition or disk clone software similar to Ghost. It saves and restores only used blocks in hard drive. By using clonezilla, you can clone a 5 GB system to 40 clients in about 10 minutes.
  • Detritus is a package of programs for aiding power users or systems administrators in finding, and as needed trimming or deleting, the variety of useless files that Windows accumulates, such as tmps, caches, logs, . . . that the MS Disk Cleanup misses.
  • openSIMS is a Security Infrastructure Management Systems that ties together the open source tools used for security event management into a common infrastructure. These tools include NMap, Snort, and many others. The best way to experience openSIMS is by downloading the openSIMS liveCD.
  • php-AD-admin will allow users to update their own AD attributes ( phonenumber etc ) and reset their own passwords. This will improve the quality of the data in your Active Directory by forms validation to create new users/groups etc.
  • PasswordNotify This tool monitors your Active Directory or OpenLDAP server via LDAP on a daily basis, and sends emails out to users who’s account are about to expire. Notification timing, messages, and frequency can all be customized with multiple profiles.
  • FWNUA (Free Windows Network User Accounting) runs silently in a Windows login script and collects data about user logins. It allows more freedom in standardized computer naming. FWNUA keeps track of the workstations so you don’t have to!
  • ManagePC is a tool for remotely administering & controlling XP/2000 machines in an Active Directory environment. It retrieves data on the hardware and software, lets you control currently running services and processes and much more.
  • Active Directory Browser (AD Browser) is a remote browser tool for viewing, managing, personalizing Microsoft(c) Server Active Directory(tm) User accounts. AD Browser could also be used as a LAN Chat application within Enterprise (Company).
  • Weyland is a tool that collects information about a switched network, presents the information in a convenient format, and displays a graph of the spanning tree of the network.
  • DAD is a Windows event log and syslog management tool that allows you to aggregate logs from hundreds to thousands of systems in real time. DAD requires no agents on the servers or workstations. Correlation and analysis is driven through a web front end.
  • SB EventLog Monitor is monitoring and consolidating Windows EventLogs. Events are collected from server using VBS and WMI or by Windows agent. Events are parsed using PHP and YOUR rules and are stored in MySQL database. Email alerts are supported.
  • Pandora FMS is a distributed monitoring and performance Free System. It supports many OS and can monitor applications, services, network daemons, SNMP, OS parameters, and much more.

Quickly Recreate Shares on Windows Servers

So you’re upgrading your server and you need to either recreate all you network shares or you’ve moved data to another drive and you want to reshare your data. With a few simple commands and a little bit of excel you can quickly make this happen.

Net Shares Command

The first thing you need to do is get a list of your current shares. So on your sever open up a command prompt and type the following: net share > c:\shares.txt

This will generate the a test file called shares.txt with all of your shares listed. The unfortunate thing is that Microsoft for some dumb reason will sometimes split the results for a single share over two lines, but that is still pretty easy to deal with.

Cleanup

Open the shares.txt file in Excel. You will need to do a little clean up to get everything the way you need it. In the end you need to just three three columns of information. The first is the share name, the second is the location of the directory to be shared and the third is the remarks and/or description field. You can also use this time to modify any of the drive locations, etc. Here is an example.

Example Excel Cleanup of Shares

Formula to Recreate Shares

So now that we have our information ready, we’re going to turn it into a DOS batch file that will create/modify our shares. The first step is use our shares information to create the necessary DOS commands. We’re going to use an Excel formula to do the heavy lifting. In this case we want to use the text in the first three columns to produce a single command that looks like this:
NET SHARE APPS=F:\Applications

The formula that you will need to place in Column D is as follows:
=IF(C2="","NET SHARE "&A2&"="&B2,"NET SHARE "&A2&"="&B2&" /REMARK:"""&C2&"""")

Now just copy and paste the formula into all of the necessary rows to complete your prep of the net shares commands.

Creating and Using the Batch File

Once you have verified all of your entries are correct. Copy the contents of your Net Shares column (Column D). Paste that into an empty text document, and change the extension to .bat. Now run your batch file while logged onto the console of the server and presto your shares have been recreated.

With a little work you can also use this script to delete and/or modify existing shares too.

Windows Server 2003 Gotchas

The way we have our network setup, we use NTFS permissions for controlling access to files, not the shares permissions. Microsoft Windows Server 2003 changed the default share behavior for the Everyone group. They removed the FULL access control on the share and changed it to READ, so in our case we just modify the formula to add the correct permission back. To do that we just tack the following to the end of the formula :&" /GRANT:Everyone,Full"

New Electronic Records Rules

I’m looking for some help from loyal readers (all 3 of you). If you were not aware, on December 1, 2006 new rules for storing and archiving electronic files went in to place. I’ve been looking for a relative concise document/article that explains in clear language what I need to do at work to insure that we are in compliance with this new regulations. I found some information at the Yale Law Journal, but it was way to legalistic for me to follow and understand.

So If you have some information and are willing to share it, please let me know. I’ll happily link to it, or even host it if need.

Like a Gunshot In the Server Room

I’m told that’s what it sounded like when the power supply in our NetApp FAS 3020 dramatically failed last week. One of my administrators got a message saying there was problem with one of NetApps when he was home during the holiday weekend. He immediately called NetApp tech support which had already opened a case for the unit (you have got to love AutoSupport). They couldn’t tell 100% which power supply it was because it apparently blew the circuit breaker in the power strip (and because the power supplies are bus powered so that even though one had died it still showed up as being “in” the system). So they sent one for the drive shelf, and my admin headed into the office.

When he arrived they determined that it probably as the power supply on the head unit and not the drive shelf. So to make sure they had him pull the head unit’s power supply, reset the circuit breaker on the power strip and then reinsert the power supply. At that point a huge spark came shooting out of the head unit’s power supply, and the circuit breaker blew in the rack’s surge suppressor, as well as the circuit breaker in the electrical panel. That’s what sounded like the gunshot. Supposedly the smell of burnt electronic was very strong as well. So they canceled the one power supply shipment and sent another one. It arrived in about 2 hours, and when swapped we were back to 100%.

But the great thing was that during the entire time the unit kept running and passing data without even one hiccup. Way to go NetApp.