Wednesday, March 30, 2016
Massive iSCSI Failure with Synology DSM 6.0
I inadvertently updated my Synology DS1515+ from DSM 5.x to 6.0 during Monday's maintenance window because I didn't pay close attention to the numbers when updating to the latest version. As a result of the update, iSCSI connections have started failing every few hours, occasionally causing the DiskStation to crash in the process.
This has disastrous consequences because the device acts as the shared iSCSI storage target for a small ESXi cluster, and hosts all of the core infrastructure services (DHCP, DNS, DC, file, print, etc.)
The only thing that seems to restore connectivity is a forced power-cycle. However, along with being very risky for potential data-loss, this has the undesirable effect of triggering a full consistency check, negatively impacting performance for nearly 24h.
User logs show no indication of any errors, but the debug logs indicate that there is a login negotiation error, despite no authentication being configured at the moment.
I'm currently working with Synology to get this resolved as quickly as possible but unfortunately they don't support downgrading to previous versions (one of the reasons I can't recommend these devices for mission critical business environments) so the fix is not as simple as reverting to 5.2.
I'll update when I have a resolution.
UPDATE 2016-04-01:
After several days of discussions with Synology, recreating iSCSI targets, and wiping/re-installing the DSM software, I have been told that this may be a bug with the iSCSI daemon in the latest release and, b/c we can't revert to an older version, there's no fix except to get Synology developers from overseas to connect from remote overnight and patch the system manually. This is not looking good!
UPDATE 2016-04-11:
Took a while to get Synology to patch the system, and I was finally able to start testing again on Friday; however, the problem has not been solved. After just over 24h of Iometer testing, the unit crashed again. I've been told there may be additional problems with iSCSI for this release.
Subscribe to:
Post Comments (Atom)

No comments:
Post a Comment