Massive iSCSI Failure with Synology DSM 6.0

I inadvertently updated my Synology DS1515+ from DSM 5.x to 6.0 during Monday's maintenance window because I didn't pay close attention to the numbers when updating to the latest version. As a result of the update, iSCSI connections have started failing every few hours, occasionally causing the DiskStation to crash in the process. This has disastrous consequences because the device acts as the shared iSCSI storage target for a small ESXi cluster, and hosts all of the core infrastructure services (DHCP, DNS, DC, file, print, etc.) The only thing that seems to restore connectivity is a forced power-cycle. However, along with being very risky for potential data-loss, this has the undesirable effect of triggering a full consistency check, negatively impacting performance for nearly 24h. User logs show no indication of any errors, but the debug logs indicate that there is a login negotiation error, despite no authentication being configured at the moment. I'm currently working with Synology to get this resolved as quickly as possible but unfortunately they don't support downgrading to previous versions (one of the reasons I can't recommend these devices for mission critical business environments) so the fix is not as simple as reverting to 5.2. I'll update when I have a resolution. UPDATE 2016-04-01: After several days of discussions with Synology, recreating iSCSI targets, and wiping/re-installing the DSM software, I have been told that this may be a bug with the iSCSI daemon in the latest release and, b/c we can't revert to an older version, there's no fix except to get Synology developers from overseas to connect from remote overnight and patch the system manually. This is not looking good! UPDATE 2016-04-11: Took a while to get Synology to patch the system, and I was finally able to start testing again on Friday; however, the problem has not been solved. After just over 24h of Iometer testing, the unit crashed again. I've been told there may be additional problems with iSCSI for this release.

Resetting the Offline Folders Database

If your Windows XP machine is trying to sync folders to a server that no longer exists, you can easily reset the Offline Folders database by adding a DWORD named FormatDatabase to HKLM\Software\Microsoft\Windows\CurrentVersion\NetCache, setting the value to 1, and rebooting the machine.

Google Apps Standard Edition

It has become increasingly difficult to find the link to the free Google Apps Standard Edition so I thought I would post it here for future reference. I hope this saves somone some time.

RealVNC on Windows Vista

RDP is a great protocol for accessing a remote Windows desktop in terms of the user experience, but one significant drawback is that you can't use it to share the screen with a local user. Not good when you're trying to explain something or gather information about a problem.

RealVNC is a great workaround for this, however, if you're using the free versiona and running Windows Vista there is a gotach - you'll have to choose not to install the VNC server as a service, and instead rely on the user to start it manually. This is due to a new security feature that restricts access to the console session. An alternate solution is to add the user-mode shortcut to the startup folder so that it automatically runs at logon. This still requires a local user to be present to log onto the machine, but I assume that you can use RDP in the other cases.

Symantec Endpoint Improving

A while ago I blogged about some less than plesant experiences with Symantec Endpoint Protection. Symantec has since made some significant improvements to the installation, configuration and administration experiences and although there are still areas that can be improved, I'm now willing to install it on many of the networks that I manage.

One of the utilities that makes the whole thing easier is the The Symantec Endpoint Protection Support Tool. This tool checks the local system configuration for a management system or a client and gives you detailed list of potential problems along with some useful links to help you solve them.

Sounds like the crew Symantec has been listening to our complaints. Good work folks!

Jotting to Blogger

I just set up my Jott account to work with Blogger. Unfortunately for you, you'll be subjected to my technology rants more often. Fortunately for you, I'm limited to only 15 seconds at a time :-)

Small Business Server 2008 Backups

I just watched Sean Daniels' video on some of the new SBS 2008 features, where he gives a sneak peek of the new backup procedure. He mentions that it is snapshot-based using VSS and geared towards (multiple) external USB/FireWire drives. This brought up a few immediate concerns for me:

1. It sounds like it could re-introduce some of the hassles that differential and incremental backups used to pose for many tape users. My understanding was that SBS tried to remove this complexity by enforcing daily full backups.
2. External drives are generally IDE/SATA based and have a relatively high failure rate (often due to poor cooling). What happens when one fails? What if it’s the one that contains your very first backup?

I'll post the answers to these questions as soon as I have them...

UPDATE: In response to my questions, Sean has been kind enough to start a FAQ here: http://sbs.seandaniel.com/2008/03/small-business-server-2008-backup-q.html.

The short version is that both of these issues have been addressed by including a full backup on each bacup drive from which the incremental backups on the same drive are calculated. Only one good drive is required for a complete restore.

SQL 2005 Express SP2

I recently had some issues installing SP2 for SQL 2005 Express on one of the servers I manage. The installation appeared to complete without incident, however, at every update check I was told that I still needed to install this critical update again.

I tried several times, expecting that my never-waning persistence would fix the problem, but no luck. I even downloaded the update and installed it manually - still no change. I reviewed the installation logs but found no errors. I verified the file versions and everything appeared to be updated. I finally resorted to calling Microsoft's Professional Support Services and with Bob's help, discovered that this was simply a matter of an incorrect registry entry.

The "Version" and "Patch" values under HKLM\SOFTWARE\Microsoft\Microsoft SQL Server\90\Tools\Setup were not updated during the SP2 installation, and were incorrectly set to 9.1.2047 rather than to 9.2.3042.00. Manually updating the values and re-running the Microsoft Update detect cycle resolved the problem.

I can't be sure whether this was the root problem, or just a symptom of something bigger, but so far everything is working as expected. Hopefully this saves someone else a few minutes.

Baffling Error Messages

I always tell my very wonderful, yet fairly computer illiterate wife to be very careful of suspicious looking messages and prompts while browsing the Internet, especially ones with poor English or unexpected requests. Being an unusually cautious user, she often asks me what to do when she encounters something she hasn't seen before. One day last week she showed me her laptop and asked me what to do with this one:

Since the only option is an OK button, my first instinct was to just go ahead and click it, but due to the strange wording of the message I was second guessing myself this time. Was this a cleverly disguised attempt to install some sort of spyware?

Only after I Googled the error on another machine did I trust that clicking the OK or close buttons would not bring the otherwise perfect system to a grinding halt. It turns out that this is simply a generic error message from Picasa Web Albums, but without any context and with the odd choice of wording, it could easily be confused for one of those "suspicious" prompts that my wife has sworn never to click on without a second opinion.

Google, if you're reading this, please consider changing this behavior to avoid adding yet another layer of confusion for the average Internet user who is already overburdened with so many other baffling messages.

Symantec Endpoint Protection 11

Frankly, I think it should be renamed Symantec Endpoint Infection until all of the remaining issues are worked out. I have spent many hours trying to make this "upgrade" to SAV 10.2 work for our small business clients, but every time I work around one issue, a new one is waiting for me just around the corner. The number of what-the-hell-were-they-thinking type problems is just plain shocking. Admitedly some of them have been fixed by the MR1 release, but even this release has it's share of issues, not the least of which is the absurd resource requirements that make it unusable on pretty much any SBS server.

But that's not what prompted tonight's rant. I was troubleshooting a networking problem shortly after what I assumed was a successful install of SEP11. The pre-MR1 version had some known DHCP related issues so one of my first steps was to disable the product to make sure it was not interfering. No dice. It turned out that the connection to my ISP was down, and resetting the modem quickly resolved this.

The following day I was having trouble opening attachments in Outlook. Since SEP11 was still disabled, and since I didn't know when I last opened an attachment on this machine I focused on Outlook and recreating profiles and re-installing MS Office. Again, no dice. After about 30 min of futzing around I pulled out my trusty Google handbook and started researching the error message. Suggestions like "reformat your PC" and "use Linux" were the norm. Not satisfied with this, I did a query for symantec endpoint outlook attachments - jackpot! Apparently MR1 introduces a problem where Outlook attachments are not available WHEN SEP11 IS TURNED OFF! Priceless. Well, at least there is a patch... to fix a maintenance release... of an upgrade.

2008-04-16 Update: MR2 has been released for download. You can find some more information about the issues that it fixes in the PDF attached to this blog entry.

Exchange Server 2007 Won't Recognize Existing Exchange 2000 Organization

I tried installing Exchange Server 2007 into an existing Exchange 2000 organization today and was continually prompted to create a new organization, even after updating the schema and switching the organization to native mode. After a little troubleshooting and reviewing the log files I found that setup could not find a suitable GC. The AD domain already has a working GC, but apparently the Windows version was too old to be considered "suitable". Making the new Exchange server - which also happens to be a Windows Server 2003 DC - a global catalog server solved the problem.

Hope that info saves someone else a little time...

UPDATE: Here's a Microsoft article describing how to do make a GC for those that need it: http://support.microsoft.com/kb/313994

Windows Vista and Metadata

In Windows XP I was able to right-click any file in Explorer and add summary information like Title, Subject, Category, Keywords, Comments directly from the property pages. I frequently used this to save document-specific information that could not or should not be saved in the document itself. For example, many of my CD backups had product names, serial numbers, website addresses etc. saved with the ISO file for later retrieval. I could quickly view this information in Explorer by displaying these additional columns.

Fast forward to Windows Vista and I can still display custom columns in the Explorer window, but the ability to enter this information is suspiciously absent. After a little Google research, I came to the following conclusions: I can still enter metadata for certain files (in fact Microsoft has even improved on this by allowing me to do it directly from the preview pane), but unfortunately this only works for file types that support metadata natively (doc, xls, mp3, etc.) Files types that don't accommodate metadata as part of the file structure (i.e. those that require an alternate data stream to save this info like txt, csv, iso, etc.) are no longer supported. And there is no alternative in sight unless I want to learn C/C++ and start writing my own property handlers. This seems like a significant step backwards to me!

In this post, Ben Betz mentions some of the reasons Microsoft decided to remove this feature. It basically boils down to the fact that (meta)data can be accidentally lost when users don't understand how it works and when programs do not take the necessary precautions during file operations.

That's just crazy! In a time when metadata is becoming so important to everything we do I find it hard to believe that the only way to combat user ignorance and poor (or perhaps only outdated) programming practices is to remove the related features. Perhaps NTFS security features should also be removed since ACL "metadata" can also be lost in many of the same ways. While we're at it we could also lower traffic fatalities if we got rid of the automobile rather than teaching people to become better drivers... but I digress.

To make matters worse, Vista not only prevents me from modifying this data, it also hides any existing alternate stream metadata from me (even though it does not actually discard it)! Try this:

• Create a new TXT file on a WinXP machine and save it.
• Right-click the file, select Properties and click the Details tab.
• Enter some text in the Comments field and click OK.
• Move the file to an NTFS network location and then to a Vista machine.
• On the Vista machine, right-click the file, select Properties and click the Details tab - notice that the comments you entered are missing.
• Move the file back to the network share, and back to the WinXP machine - notice your comments appear again.

And to make things even more confusing, when I add the Comments column to the details view in Explorer, only WinXP displays the metadata I saved earlier. Vista shows me an empty column implying that my file does NOT contain this data. How exactly does this help me from losing my precious metadata? Incidentally, I'm curious as to how the indexing service handles all of this.

I really hope that Microsoft's decision to remove support for metadata in alternate data streams was really a matter of not getting the upgraded/fixed feature finished in time for Vista's release date, and not a permanent change. Maybe I'll get lucky with Vista SP1. Until then, I'll be forced to keep a WinXP machine around just for this purpose.

Universal Feature Wishlists (UFWs)

In my brief quest to find the ultimate blogging and photo sharing combination I found many excellent applications that either have a deal-breaking flaw or require some kludge of a workaround to fit them into a particular workflow. More often than not, my search for an elegant solution only turned up more people with the same unresolved issues. After several days of searching for perfection, I had a bit of a revelation - every piece of software and every web application does certain things well, and others not so well.

Obvious, perhaps, but it made me question why nobody has created a Universal Feature Wishlist (UFW) - some kind of categorized, wiki-style poll database where anyone can add new ideas for functionality, vote for existing ones or give kudos to those that have implemented something brilliantly.

For example, I think that online photo albums like Picasa Web Albums should have a template system to personalize their look, and to allow for the kind of scripting required by Google Analytics. I also think that the template and layout interface used on Blogger is excellent - it allows for easy point-and-click editing as well as direct modification of the HTML code. Marrying the two would be brilliant!

I suppose I could send the vendor a feature request, but the likelihood of getting it implemented is probably pretty low. However, I suspect if there was an established UFW, your chances would increase dramatically since the competition would likely be working on the same thing too. Vendors would start competing over the quality of the features, rather than the quantity, and we would eventually end up with fewer bugs and faster code. And, if open source developers and hobbyists had access to this kind of information, we would probably see a lot more of those rare applications that make you think "Wow, this is exactly what I was looking for!"

Fantastic. Now that I've solved all the world's software problems I just need to find a UFW to determine the features that will be included in the first UFW...