Massive iSCSI Failure with Synology DSM 6.0

I inadvertently updated my Synology DS1515+ from DSM 5.x to 6.0 during Monday's maintenance window because I didn't pay close attention to the numbers when updating to the latest version. As a result of the update, iSCSI connections have started failing every few hours, occasionally causing the DiskStation to crash in the process. This has disastrous consequences because the device acts as the shared iSCSI storage target for a small ESXi cluster, and hosts all of the core infrastructure services (DHCP, DNS, DC, file, print, etc.) The only thing that seems to restore connectivity is a forced power-cycle. However, along with being very risky for potential data-loss, this has the undesirable effect of triggering a full consistency check, negatively impacting performance for nearly 24h. User logs show no indication of any errors, but the debug logs indicate that there is a login negotiation error, despite no authentication being configured at the moment. I'm currently working with Synology to get this resolved as quickly as possible but unfortunately they don't support downgrading to previous versions (one of the reasons I can't recommend these devices for mission critical business environments) so the fix is not as simple as reverting to 5.2. I'll update when I have a resolution. UPDATE 2016-04-01: After several days of discussions with Synology, recreating iSCSI targets, and wiping/re-installing the DSM software, I have been told that this may be a bug with the iSCSI daemon in the latest release and, b/c we can't revert to an older version, there's no fix except to get Synology developers from overseas to connect from remote overnight and patch the system manually. This is not looking good! UPDATE 2016-04-11: Took a while to get Synology to patch the system, and I was finally able to start testing again on Friday; however, the problem has not been solved. After just over 24h of Iometer testing, the unit crashed again. I've been told there may be additional problems with iSCSI for this release.

Resetting the Offline Folders Database

If your Windows XP machine is trying to sync folders to a server that no longer exists, you can easily reset the Offline Folders database by adding a DWORD named FormatDatabase to HKLM\Software\Microsoft\Windows\CurrentVersion\NetCache, setting the value to 1, and rebooting the machine.

Google Apps Standard Edition

It has become increasingly difficult to find the link to the free Google Apps Standard Edition so I thought I would post it here for future reference. I hope this saves somone some time.

RealVNC on Windows Vista

RDP is a great protocol for accessing a remote Windows desktop in terms of the user experience, but one significant drawback is that you can't use it to share the screen with a local user. Not good when you're trying to explain something or gather information about a problem.

RealVNC is a great workaround for this, however, if you're using the free versiona and running Windows Vista there is a gotach - you'll have to choose not to install the VNC server as a service, and instead rely on the user to start it manually. This is due to a new security feature that restricts access to the console session. An alternate solution is to add the user-mode shortcut to the startup folder so that it automatically runs at logon. This still requires a local user to be present to log onto the machine, but I assume that you can use RDP in the other cases.

Symantec Endpoint Improving

A while ago I blogged about some less than plesant experiences with Symantec Endpoint Protection. Symantec has since made some significant improvements to the installation, configuration and administration experiences and although there are still areas that can be improved, I'm now willing to install it on many of the networks that I manage.

One of the utilities that makes the whole thing easier is the The Symantec Endpoint Protection Support Tool. This tool checks the local system configuration for a management system or a client and gives you detailed list of potential problems along with some useful links to help you solve them.

Sounds like the crew Symantec has been listening to our complaints. Good work folks!

Jotting to Blogger

I just set up my Jott account to work with Blogger. Unfortunately for you, you'll be subjected to my technology rants more often. Fortunately for you, I'm limited to only 15 seconds at a time :-)

Small Business Server 2008 Backups

I just watched Sean Daniels' video on some of the new SBS 2008 features, where he gives a sneak peek of the new backup procedure. He mentions that it is snapshot-based using VSS and geared towards (multiple) external USB/FireWire drives. This brought up a few immediate concerns for me:

1. It sounds like it could re-introduce some of the hassles that differential and incremental backups used to pose for many tape users. My understanding was that SBS tried to remove this complexity by enforcing daily full backups.
2. External drives are generally IDE/SATA based and have a relatively high failure rate (often due to poor cooling). What happens when one fails? What if it’s the one that contains your very first backup?

I'll post the answers to these questions as soon as I have them...

UPDATE: In response to my questions, Sean has been kind enough to start a FAQ here: http://sbs.seandaniel.com/2008/03/small-business-server-2008-backup-q.html.

The short version is that both of these issues have been addressed by including a full backup on each bacup drive from which the incremental backups on the same drive are calculated. Only one good drive is required for a complete restore.

SQL 2005 Express SP2

I recently had some issues installing SP2 for SQL 2005 Express on one of the servers I manage. The installation appeared to complete without incident, however, at every update check I was told that I still needed to install this critical update again.

I tried several times, expecting that my never-waning persistence would fix the problem, but no luck. I even downloaded the update and installed it manually - still no change. I reviewed the installation logs but found no errors. I verified the file versions and everything appeared to be updated. I finally resorted to calling Microsoft's Professional Support Services and with Bob's help, discovered that this was simply a matter of an incorrect registry entry.

The "Version" and "Patch" values under HKLM\SOFTWARE\Microsoft\Microsoft SQL Server\90\Tools\Setup were not updated during the SP2 installation, and were incorrectly set to 9.1.2047 rather than to 9.2.3042.00. Manually updating the values and re-running the Microsoft Update detect cycle resolved the problem.

I can't be sure whether this was the root problem, or just a symptom of something bigger, but so far everything is working as expected. Hopefully this saves someone else a few minutes.

Baffling Error Messages

I always tell my very wonderful, yet fairly computer illiterate wife to be very careful of suspicious looking messages and prompts while browsing the Internet, especially ones with poor English or unexpected requests. Being an unusually cautious user, she often asks me what to do when she encounters something she hasn't seen before. One day last week she showed me her laptop and asked me what to do with this one:

Since the only option is an OK button, my first instinct was to just go ahead and click it, but due to the strange wording of the message I was second guessing myself this time. Was this a cleverly disguised attempt to install some sort of spyware?

Only after I Googled the error on another machine did I trust that clicking the OK or close buttons would not bring the otherwise perfect system to a grinding halt. It turns out that this is simply a generic error message from Picasa Web Albums, but without any context and with the odd choice of wording, it could easily be confused for one of those "suspicious" prompts that my wife has sworn never to click on without a second opinion.

Google, if you're reading this, please consider changing this behavior to avoid adding yet another layer of confusion for the average Internet user who is already overburdened with so many other baffling messages.

Symantec Endpoint Protection 11

Frankly, I think it should be renamed Symantec Endpoint Infection until all of the remaining issues are worked out. I have spent many hours trying to make this "upgrade" to SAV 10.2 work for our small business clients, but every time I work around one issue, a new one is waiting for me just around the corner. The number of what-the-hell-were-they-thinking type problems is just plain shocking. Admitedly some of them have been fixed by the MR1 release, but even this release has it's share of issues, not the least of which is the absurd resource requirements that make it unusable on pretty much any SBS server.

But that's not what prompted tonight's rant. I was troubleshooting a networking problem shortly after what I assumed was a successful install of SEP11. The pre-MR1 version had some known DHCP related issues so one of my first steps was to disable the product to make sure it was not interfering. No dice. It turned out that the connection to my ISP was down, and resetting the modem quickly resolved this.

The following day I was having trouble opening attachments in Outlook. Since SEP11 was still disabled, and since I didn't know when I last opened an attachment on this machine I focused on Outlook and recreating profiles and re-installing MS Office. Again, no dice. After about 30 min of futzing around I pulled out my trusty Google handbook and started researching the error message. Suggestions like "reformat your PC" and "use Linux" were the norm. Not satisfied with this, I did a query for symantec endpoint outlook attachments - jackpot! Apparently MR1 introduces a problem where Outlook attachments are not available WHEN SEP11 IS TURNED OFF! Priceless. Well, at least there is a patch... to fix a maintenance release... of an upgrade.

2008-04-16 Update: MR2 has been released for download. You can find some more information about the issues that it fixes in the PDF attached to this blog entry.

Exchange Server 2007 Won't Recognize Existing Exchange 2000 Organization

I tried installing Exchange Server 2007 into an existing Exchange 2000 organization today and was continually prompted to create a new organization, even after updating the schema and switching the organization to native mode. After a little troubleshooting and reviewing the log files I found that setup could not find a suitable GC. The AD domain already has a working GC, but apparently the Windows version was too old to be considered "suitable". Making the new Exchange server - which also happens to be a Windows Server 2003 DC - a global catalog server solved the problem.

Hope that info saves someone else a little time...

UPDATE: Here's a Microsoft article describing how to do make a GC for those that need it: http://support.microsoft.com/kb/313994

Windows Vista and Metadata

In Windows XP I was able to right-click any file in Explorer and add summary information like Title, Subject, Category, Keywords, Comments directly from the property pages. I frequently used this to save document-specific information that could not or should not be saved in the document itself. For example, many of my CD backups had product names, serial numbers, website addresses etc. saved with the ISO file for later retrieval. I could quickly view this information in Explorer by displaying these additional columns.

Fast forward to Windows Vista and I can still display custom columns in the Explorer window, but the ability to enter this information is suspiciously absent. After a little Google research, I came to the following conclusions: I can still enter metadata for certain files (in fact Microsoft has even improved on this by allowing me to do it directly from the preview pane), but unfortunately this only works for file types that support metadata natively (doc, xls, mp3, etc.) Files types that don't accommodate metadata as part of the file structure (i.e. those that require an alternate data stream to save this info like txt, csv, iso, etc.) are no longer supported. And there is no alternative in sight unless I want to learn C/C++ and start writing my own property handlers. This seems like a significant step backwards to me!

In this post, Ben Betz mentions some of the reasons Microsoft decided to remove this feature. It basically boils down to the fact that (meta)data can be accidentally lost when users don't understand how it works and when programs do not take the necessary precautions during file operations.

That's just crazy! In a time when metadata is becoming so important to everything we do I find it hard to believe that the only way to combat user ignorance and poor (or perhaps only outdated) programming practices is to remove the related features. Perhaps NTFS security features should also be removed since ACL "metadata" can also be lost in many of the same ways. While we're at it we could also lower traffic fatalities if we got rid of the automobile rather than teaching people to become better drivers... but I digress.

To make matters worse, Vista not only prevents me from modifying this data, it also hides any existing alternate stream metadata from me (even though it does not actually discard it)! Try this:

• Create a new TXT file on a WinXP machine and save it.
• Right-click the file, select Properties and click the Details tab.
• Enter some text in the Comments field and click OK.
• Move the file to an NTFS network location and then to a Vista machine.
• On the Vista machine, right-click the file, select Properties and click the Details tab - notice that the comments you entered are missing.
• Move the file back to the network share, and back to the WinXP machine - notice your comments appear again.

And to make things even more confusing, when I add the Comments column to the details view in Explorer, only WinXP displays the metadata I saved earlier. Vista shows me an empty column implying that my file does NOT contain this data. How exactly does this help me from losing my precious metadata? Incidentally, I'm curious as to how the indexing service handles all of this.

I really hope that Microsoft's decision to remove support for metadata in alternate data streams was really a matter of not getting the upgraded/fixed feature finished in time for Vista's release date, and not a permanent change. Maybe I'll get lucky with Vista SP1. Until then, I'll be forced to keep a WinXP machine around just for this purpose.

Universal Feature Wishlists (UFWs)

In my brief quest to find the ultimate blogging and photo sharing combination I found many excellent applications that either have a deal-breaking flaw or require some kludge of a workaround to fit them into a particular workflow. More often than not, my search for an elegant solution only turned up more people with the same unresolved issues. After several days of searching for perfection, I had a bit of a revelation - every piece of software and every web application does certain things well, and others not so well.

Obvious, perhaps, but it made me question why nobody has created a Universal Feature Wishlist (UFW) - some kind of categorized, wiki-style poll database where anyone can add new ideas for functionality, vote for existing ones or give kudos to those that have implemented something brilliantly.

For example, I think that online photo albums like Picasa Web Albums should have a template system to personalize their look, and to allow for the kind of scripting required by Google Analytics. I also think that the template and layout interface used on Blogger is excellent - it allows for easy point-and-click editing as well as direct modification of the HTML code. Marrying the two would be brilliant!

I suppose I could send the vendor a feature request, but the likelihood of getting it implemented is probably pretty low. However, I suspect if there was an established UFW, your chances would increase dramatically since the competition would likely be working on the same thing too. Vendors would start competing over the quality of the features, rather than the quantity, and we would eventually end up with fewer bugs and faster code. And, if open source developers and hobbyists had access to this kind of information, we would probably see a lot more of those rare applications that make you think "Wow, this is exactly what I was looking for!"

Fantastic. Now that I've solved all the world's software problems I just need to find a UFW to determine the features that will be included in the first UFW...

Replacing Notepad on Windows Vista with UAC Enabled

One of the first things I do after setting up a new Windows machine is replace the built-in notepad application. My favorite replacement is the lightweight and free Notepad2 by Florian Balmer. In Windows XP it's basically a matter of replacing all the copies of the notepad.exe file and ignoring the Windows File Protection prompt to "fix" the situation. I have even created a simple installer to do the grunt work for me.

In Windows Vista, however, it's not so easy. The file replacement works as expected but every time Notepad2 starts Windows shows the Security Warning: "The publisher could not be verified. Are you sure you want to run this software?" In Windows XP you can simply unchecked the option to "Always ask before opening this file", or click the Unblock button in one of the executable's property pages. However, the combination of Vista's User Account Control (UAC) and IE7's Persistent Zone Identifier makes this impossible from the standard GUI. Neither of these changes remains permanent, and the next time you start Notepad2 the warning is displayed agin. Needles to say, this gets bothersome very quickly if you're like me and are trying to avoid turning off UAC altogether.

Recently I found this post on how to use a cool little utility called runmenu to display an executable's property page with elevated rights. Using the same method I was able to unblock Notepad2 permanently and now I'm no longer bothered with the anoying warnings.

2008-02-01: UPDATE: I have found another issue with replacing the original notepad.exe - recently I have had to take ownership of the original file and then grant the local Administrators group full control before being able to replace the notepad.exe file. It seems that only the TrustedInstaller account has these rights by default. I don't remember having to do this the first few times so it may mean there has been a security change in one of the updates..

Spring Cleaning

Okay, so maybe it's a little too late to call it "spring" cleaning, but I figured it was about time to purge a lot of old posts, many of which were just personal bookmarks and links to other peoples articles. I'm hoping that this will help me get started with posting some of my own material... we'll see.

I'm also trying to figure out whether I should use the combination of Google's Blogger + Picasa Web Albums or the free version of Cummunity to host my blog, photos, files etc. Any comments would be more than welcome.

Network Configuration Profiles

Have you ever needed an easy way to switch between multiple network configurations? Are third party utilities too clumsy, limited or intrusive? The following article outlines an easy method that uses functionality already included with Windows.

As an network consultant and administrator, I have configured my laptop with my favorite tools for diagnosing problems and doing routine maintenance at various customers' sites. In order to use many of these tools, I often have to change my network configuration to work with the particular LAN I am connected to. Manually changing the relevant settings (IP address, DNS servers, Gateways etc.) turned out to be a time consuming process that I did not want to live with. After several unsuccessful searches for a pre-packaged solution, I decided to try and automate the process as much as I could using features already built into windows. I eventually found a solution using only the Netsh command and file associations.

Gathering Settings
The first step in creating network configuration profiles is to capture the settings for LAN you are connecting to. First I'll need to configure my network adapter with the appropriate TCP/IP settings and verify that they work. When I have verified that everything works, I can save my settings to a file by typing the following at a command prompt:

netsh -c interface dump > profile_name.ncp

Notice that I chose the extension .ncp for my configuration file - an extension that is not already in use on my system. This way I can configure windows to apply my saved settings using a simple file association. You are free to chose any extension you like, as long as it's not already associated to another program. You should perform this step for each different network configuration you plan to use.

Applying Saved Settings
In order to re-apply the settings to my system, I can type the following at a command prompt:

netsh exec profile_name.ncp

Netsh will apply all the settings in my previously saved configuration file to the system. Now I can easily apply different settings for each client network that I connect to.

Creating a File Association
Because I'm really lazy, I would like to simplify the previous step even more. I can do this by creating a simple file association in Windows. This will allow me to simply double-click the settings file and have the network adapter updated automatically. Here are the steps I need to take to do this:

• Open Folder Options from the Control Panel and select the File Types tab
• Click New to create a new entry and type NCP in the file extension box. Click OK to add the file type.
• Under Registered file types, select NCP and click the Advanced button to edit the definition.
• Optionally, enter a description of the file type (ex: Netsh Config File).
• Optionally, change the icon for this file type to easily identify your configuration files.
• Click New to create a new action and enter a name (ex: Apply Network Settings).
• Enter the following text in the Application used to perform action box: "C:\WINDOWS\system32\netsh.exe" exec "%1"
• Select the Use DDE checkbox and type netsh in the Application field and System in the Topic field.
• Click OK to add the action.

That's it. Now a simple double click of the configuration file will apply the settings to my system. By repeating the last four steps above with some slight modifications (hint: use notepad.exe) I can easily add an Edit action to the right click menu of my configuration files - an exercise I will leave up to the reader.

Special thanks go out to Daniel Petri for his great article on Configuring TCP/IP from the Command Prompt.

TrayMonitor v1.1.0

TrayMonitor is a small VB.NET utility that provides a quick overview of performance counters from the system tray.

Each running instance displays a small bar graph corresponding to a performance counter that you select. The results are averaged over a configurable number of samples to provide a smoother animation. Update frequency, colors and maximum values are configurable, and the settings can be saved and loaded from simple, editable configuration files.

By associating the file extension (TMC) with the executable you can quickly start your favorite counters with a double-click, or even add them to your Startup folder so they are running each time you log on.

This project was a bit of a learning exercise for me, and a lot of the code has been borrowed/modified from the ConfigOpt and NotifyIconChart articles at http://www.codeproject.com/ and other places on the w3

You can download the file here, or if you would like a copy of the source code, please send me a quick email.