Author Archives: mesabigroup

IBM Advances Cluster Virtualization for HPC and Deep Learning AI

On the classic Groucho Marx quiz show You Bet Your Life if a contestant accidently said the “secret word” of the day, he or she would win a prize. There’s no prize included in this commentary, but the secret word of the day is virtualization, especially as it relates to IBM’s new HPC and AI solutions.

IBM defines virtualization as “A technology that makes a set of shared resources appear to each workload as if they were dedicated only to it.” IT is very familiar with this concept, what with operating system-level virtualization, server virtualization, network virtualization, and storage virtualization all continuing to permeate more and more through computing infrastructures and the collective consciousness. So, it should come as no surprise that IBM is advancing the concept of cluster virtualization in its latest announcement, tying it closely to cloud and cognitive computing.

IBM’s cluster virtualization initiative combines products from its Spectrum Computing family, namely Spectrum LSF, Spectrum Symphony, and Spectrum Conductor, along with overall cluster virtualization software (Spectrum Cluster Foundation) to manage the whole process. And that includes the storage that is delivered through IBM Spectrum Scale, another member of the IBM Spectrum Storage family.

The goal of this approach is to automate the self-service provisioning of multiple heterogeneous high-performance computing (HPC) and analytics (AI and big data) clusters on a shared secure multi-tenant compute and storage infrastructure. Doing so delivers multiple benefits to numerous technical computing end users, including data scientists and HPC professionals.

The announcement focuses on these products: IBM Spectrum LSF, IBM Spectrum Conductor, and IBM Spectrum Scale.

IBM Spectrum LSF Suites 10.2

IBM Spectrum LSF is the latest release of an end-to-end workload management solution for HPC. It enables multiple users and applications to share a computing cluster, thus optimizing performance and cost efficiencies while simplifying resource management. Spectrum LSF comes in three versions: 1) The Workgroup Edition is no slouch as it can deal with up to 128 nodes and 25,000 jobs, 2) The HPC Edition scales to 1024 nodes and 250,000 jobs while adding workflow automation, hybrid cloud auto-scaling, and intelligent data staging, and 3) The Enterprise Edition takes off all scaling limits on nodes and jobs. IBM states that Spectrum LSF Suite 10.2 offers more functionality at a lower cost through a new licensing and pricing model.

IBM Spectrum Conductor

Apache Spark is an open-source cluster computing framework that provides a fast, general engine for big data processing, including built-in modules for machine learning, SQL and streaming. IBM is a strong supporter of Spark, which is why it positioned this product initially as IBM Spectrum Conductor with Spark.  While the new version continues to include the Apache Spark framework, IBM has simplified the name to IBM Spectrum Conductor to avoid confusion that it is for Spark-based workloads only.

IBM Spectrum Conductor enables an enterprise to establish and manage multiple Spark deployments while eliminating inefficient resource silos tied to multiple Apache Spark implementations. It also extends beyond Spark and eliminates cluster sprawl through a shared resource pool, along with granular and dynamic resource allocation that improves both performance and resource usage. The IBM-supported distribution also has workload, resource and data management capabilities that, along with high-performance shared storage through IBM Spectrum Scale, delivers an enterprise-class solution.

A key focus of IBM’s approach is on multitenant shared services/resources where logically each group appears to have its own Spark cluster, which provides isolated, protected, secured and SLA-managed resource allocation. The overall result, though, is being able to dynamically respond to changing workload demand. IBM claims this leads to a 43% reduction in required infrastructure over a siloed infrastructure.

The biggest news is an add-on called IBM Spectrum Conductor Deep Learning Impact. Deep learning is a member of the broader family of machine learning methods that are used to train and extend AI (artificial intelligence) applications and services.

The software extension for Spectrum Conductor enables enterprises to build distributed environments for simplifying deep learning (using pre-built frameworks, such as Caffe and TensorFlow), accelerating time-to-results (more accurate models faster), and easing administration and management processes.

IBM Spectrum Scale 5.0

HPC and deep learning AI are heavily data-driven, so storage and compute resources work seamlessly in cluster virtualization implementations. IBM supports these capabilities with its Spectrum Scale available as software, on public clouds or software embedded with the IBM Elastic Storage Server (ESS). Spectrum Scale is the file-oriented member of IBM Spectrum Storage, which is a family of software-defined storage (SDS) products that deal with immense volumes of semi-structured and unstructured data commonly used in HPC, big data and AI applications. Spectrum Scale uses the GPFS (general-purpose file system), which is a long-proven, robust technology, at its core.

IBM’s next generation ESS combines the clustered file system of Spectrum Scale with the CPU and I/O capability of IBM’s POWER architecture and connects to solid-state drives (SSDs) for performance-related requirements or just a bunch of disk (JBOD) arrays for lower cost/capacity-related requirements. ESS uses a building block approach with a minimum configuration of two IBM Power System servers and at least one storage enclosure. ESS starts with a minimum capacity of 40 TB, but can scale into hundreds of petabytes.

The objective of the latest release, Spectrum Scale 5.0, is to meet the requirements of the CORAL (Collaboration of Oak Ridge, Argonne, and Livermore national laboratories) to advance leading edge HPC technologies. ESS running Spectrum Scale V5 meets performance goals of CORAL (as stated in a CORAL RFP), such as 1 TB/sec 1MB sequential read/write and 50K creates/sec per shared directory. These capabilities are the result of improvements, including significantly reduced latency between nodes (to support NVMe), an enhanced GUI to improve system administration and improved security and compliance functionality with integrated file audit logging.

Mesabi musings

These new solutions demonstrate how IBM has built a cluster virtualization approach using products from its Spectrum Computing family — namely Spectrum LSF for HPC and Spectrum Conductor with its Deep Learning Impact add-on for deep learning AI.  Cluster virtualization enables very large numbers of users and applications to share common pools of resources, including the massive implementations planned by CORAL.

Even though there is no monetary prize for the secret word of the day — virtualization — better understanding IBM’s cluster virtualization approach offers benefits of its own. And if you are a member of the HPC or deep learning AI communities, IBM’s new cluster virtualization solutions qualify as a very big prize, indeed.

IBM Continues to Drive Forward with FlashSystem All-Flash Storage

IBM’s new storage-related announcements focus heavily on the FlashSystem portion of its all-flash storage portfolio which has as its foundation the FlashSystem 900. That includes the FlashSystem 900 itself, a standalone 2U rack solution for tier 0 application acceleration, the FlashSystem V9000 for traditional tier 1 enterprise-class applications, and the FlashSystem A9000/FlashSystem A9000R for cloud-based, enterprise-class applications.

The “R” stands for Rack and the A9000R is the biggest brother in the A9000 family, offering greater maximum capacity compared to its smaller sibling. Both can be incorporated as part of a VersaStack solution, which you will recall is Cisco and IBM’s name for its converged infrastructure (CI) solution that combines servers, storage and networking into a pre-configured bundle.

This announcement also continues IBM’s trend to emphasize its all-flash storage systems. If you would like additional background on IBM’s all-flash portfolio and strategy, see .

IBM’s announcement highlights major enhancements to the FlashSystem 900 which all members of the FlashSystem family inherit. Those other family members also have their own new functions, features and upgrade capabilities. Let’s take a closer look at them all.

IBM FlashSystem 900

  • Move From 2D to 3D Flash Technology. The most dramatic change is in the flash media itself, shifting from the use of 2D planar NAND MLC (multi-level cell) flash to 3D NAND TLC (triple-level cell) flash. This move takes advantage of die stacking to increase storage density, which is how the maximum usable capacity of the systems has tripled from 57 TB to 180 TB in the same 2U rack. TLC flash actually has a higher endurance than planar MLC, which is why IBM has no trouble offering a 7-year flash wear guarantee for its new solutions. The low latency, high IOPS, and strong bandwidth are the same as before with the data reduction feature’s inline hardware compression turned on.
  • Inline Hardware Compression. Sticking with that subject, inline hardware compression is now being offered for the first time on the FlashSystem 900. The maximum effective capacity is now 220 TB, which is a 4X (instead of the 3X improvement without it) over the 2D version’s maximum usable capacity. Plus, the compression/decompression algorithm is completely implemented in hardware. Since no processor intervention is required, there is no performance impact on server functions. Compression and decompression are performed in real-time with minimal added latency to inbound and outbound data flows. Overall performance continues (as it does without compression) to scale linearly.
  • New User Interface to Manage System. The FlashSystem 900 now uses a graphical-user interface (GUI) instead of a command-line interface (CLI). The GUI’s consolidated dashboard shows a system’s performance, current activity and active events on a single screen. The interface also enables the setting of custom notifications, such as when a system’s effective capacity reaches a pre-specified level. Quite frankly, a GUI that clearly shows a consolidated view of the three critical storage metrics of performance, capacity, and health enables storage administrators with less training to manage more storage with better results. In turn, that can also lead to fewer storage administrators being required for a given amount of managed storage. This results in an OPEX decrease with improvements in operational efficiency that go hand in glove with the CAPEX improvements that the move to 3D flash and compression enable.

IBM FlashSystem A9000/FlashSystem A9000R

IBM’s FlashSystem A9000/FlashSystem A9000R use the company’s new 3D TLC NAND MicroLatency modules, (see for more details). The company states that the modules offers up to a 200% capacity increase and also claims a 66% reduction in space, cooling and management cost per TB in the A9000/FlashSystem A9000R. The new solutions can also asynchronously mirror to a disaster recovery site that uses IBM XIV Gen 3 systems with hard disk drives. This is important since utilizing HDDs significantly reduces disaster recovery (DR) costs.

IBM FlashSystem V9000

The IBM FlashSystem V9000 introduces a new AE3 flash enclosure that using the 3D TLC technology. A customer’s IT managers can choose between what IBM calls “ultra-performance” for meeting software service-level agreement (SLA) latency, IOPS and bandwidth requirements or capacity-optimized functionality for dealing with increasing data volumes and overall storage capacity.

On the ultra-performance side, either 3.6 TB or 8.5TB MicroLatency modules can use the 2:1 no-performance-impact inline hardware compression discussed in the section on the FlashSystem 900. IBM states that this can result in a 50% overall performance improvement of up to 5.2M IOPS per system.

On the capacity-optimized side, the system uses 18 TB modules using a different compression technology called Data Reduction Pools (DRP). DRP is a combination of hardware-assisted block level and the SCSI UNMAP advisory command that is used to reclaim blocks of data that have been logically deleted, but not physically deleted. IBM stated that DRP can lead to up to a 5:1 level of data reduction. The usual caveat of “your mileage may vary” applies, but IBM also offers a Comprestimator software tool for estimating block compression rates so customers can better understand the potential benefits.  The company also believes that up to 80% of all workloads can be compressed to some extent.

Combining the 3X capacity improvement due to using 3D flash and the DRP algorithm means that a single enclosure can have up to 900 TB effective capacity and that a single system with multiple enclosures can scale up to an effective capacity of 7.2 PB.

IBM Block Storage Container Support

Docker container environments are becoming increasingly popular, but they often need persistent storage that remains intact unless administrators choose to change or delete it. IBM now offers persistent storage for Docker container environments using orchestration with IBM Cloud Private and Kubernetes. IBM supports these solutions in all storage offerings built on IBM Spectrum Virtualize and IBM Spectrum Accelerate, as well as VersaStack. That includes the FlashSystem A9000/A9000R and the FlashSystem V9000. Spectrum Virtualize also supports many other IBM and non-IBM storage systems (over 400 in all).

A key benefit of this approach is to manage cloud-native operations using stateful containers based on IBM storage. For example, the cloud-optimized IBM FlashSystem A9000/A9000R can apply end-to-end cloud economics that reduce costs for Docker container environments by simplifying complexity, ensuring necessary high-performance consistency, and taking advantage of grid scalability.

Mesabi musings

All-flash storage systems continue to attract more and more attention as they become the primary storage vehicle of choice for tier 1 production storage, replacing traditional HDD-based systems. And IBM continues to be a mover and shaker in driving that trend as revealed in its latest all-flash storage announcement.

The first major enhancement in these new solutions is the introduction of 3D NAND TLC technology to supplant the established 2D planar NAND MLC technology, resulting in a 3X increase in maximum native usable capacity. The second is the introduction of compression technology that increases the maximum effective capacity to a higher level than the maximum native usable capacity.

Along with other enhancements, such as the new user-friendly GUI, the net result is that IBM is offering significantly improved performance as well as better CAPEX and OPEX results. That should result in FlashSystem solutions becoming even more attractive choices for IBM customers.

Altaro Deliver Virtual Machine Backup for Small and Mid-Market Companies

Altaro provides virtual machine (VM) backup for small and mid-market (SME) companies (those with less than a thousand employees) that typically cannot afford the deep and robust IT organizations that large enterprises have at their disposal. Still, these businesses must keep up with the latest IT initiatives, including the move to virtual machines (VMs) (notably using Microsoft’s Hyper-V and VMware solutions). Accompanying these moves is the need for VM storage backup to provide the necessary levels of data protection that these businesses require. And that is where Altaro comes in.

The SME VM backup market is a strong one with competitors, such as Switzerland-based Veeam and Massachusetts-based Acronis, having made strong names for themselves. However, Malta-based Altaro does not take a back seat to anyone, posting 40% year over year growth and 40,000 business customers that use company solutions, including the recently introduced Altaro VM Backup V7. 

Altaro is proud of its inline deduplication introduced earlier this year that it claims supports a better deduplication ratio than any of its competitors. It also offers data backup health checker that flags any corruption noted from a failed checksum, then takes the appropriate corrective actions. Altaro’s core values include ease of use and technical support. Yes, the necessary technology has to be there, but without those two core values IT-resource-constrained SMEs would have trouble with any application. The company’s large client base and customer testimonials verify that Altaro really delivers what it promises.

But the IT resource limitations of SMEs also imply that they often need to turn to someone else for different aspects of using software products, such as installation, deployment and operation. Altaro’s partners can provide customers with as little or as much support as needed.

In addition, many SMEs are looking into the use of the cloud and turning to a managed service provider (MSP) to provide backup as a subscription service. That is why Altaro is now offering Altaro VM Backup for MSPs. An MSP can offer a business a monthly subscription to manage its backups. The MSP uses the Altaro Cloud Management Console (CMC) to manage all the necessary processes for all customers (i.e., multi-tenancy) through a single interface.

That of course means managing all backup and restores, but there are other key capabilities, as well. For planning purposes, Altaro CMC can create templates for backup schedules and retention policies. For operational purposes, the CMC can receive real-time updates and alerts that can help dictate if any action is required by the MSP CMC administrator. Moreover, the MSP can operate CMC through a Web browser, thus eliminating any need to use a VPN or to be on-site.

An MSP can currently use Microsoft Azure as the target cloud for Altaro, but the company intends to add support for Amazon Web Services (AWS) and Google Cloud in the future.

Mesabi musings

Just because they are not enterprise-class in number of employees and IT resources does not mean that SMEs do not need robust and reliable data protection for their VM environments, Altaro’s VM Backup addresses this need. By emphasizing ease of use and technical support, the company makes life easier for its customers whether they do everything on their own or need help from Altaro partners or decide to do backup through a subscription model with an MS. You don’t have to be a big company to need data protection, and Altaro focuses on providing those businesses the services and solutions they require.

Dell EMC Strengthens Data Protection for the Software-Defined Data Center

Dell EMC has long demonstrated its leadership in data protection and continues to extend that role with data protection offerings for the software-defined data center (SDDC) — notably, those data centers under the aegis of the VMware family of products.

Dell EMC Data Protection Suite Family

But before we get into the company’s latest product announcements that came this week at VMworld 2017, we should review a major focal point of the Dell EMC Data Protection Division, namely the Dell EMC Data Protection Suite Family of software products. These family members emphasize who they are for (e.g. customer-focused) or what they do (e.g., purpose-focused) rather than the brand names once associated with them.

This is important, as a particular solution may blend or combine software products and can be tailored to meet the specific needs of an IT organization. For example, Dell EMC’s Avamar and NetWorker are among the many software products available in the Enterprise Edition. Moreover, an original software product might be used in more than one product. A case in point: NetWorker is available for both the Enterprise Edition and the Backup family members.

The Dell EMC Data Protection Suite Family offers features/functions, including:

  • Data Protection Suite Enterprise Edition — with the Enterprise Edition, IT organizations have access to all of Dell EMC’s software products, including those for continuous replication, snapshots, traditional backup and archive.
  • Data Protection Suite for Backup — traditional backup and recovery including disk-based backup, snapshot-based backup, and backup to tape
  • Data Protection Suite for Applications — the focus is on ensuring the meeting of stringent service level agreement data protection objectives for mission-critical applications through such capabilities as copy data management and direct backup to Data Domain storage.
  • Data Protection Suite for VMware — data protection for VMware-based environments, including backup and recovery and continuous replication as well as monitoring, analysis, and search capabilities are the focus of this suite family member.
  • Data Protection Suite for Archive — archiving to save operational costs (such as reclaiming primary data storage) as well as serving compliance and regulatory needs, such as serving as the basis for e-discovery is the purpose of this suite family member.

Data protection for the software-defined data center

The SDDC is a transformation from how business was supported by traditional data centers and their collections of legacy applications. Instead, the modern SDDC features IT infrastructure assets that are physically- service-, and IT operational software-related. On the physical side, the move is toward converged/hyperconverged infrastructures. On the service side, the infrastructure is increasingly being delivered as a service (IaaS). In addition, the infrastructure management software is becoming increasingly automated.


Dell EMC stresses that automation is especially important for the infrastructure that supports data protection for VMware-based data centers. The company believes that it is the only vendor that automates VM backup that includes not only policy management, but also data movers/proxies and protection storage. That architecture matters in order to provide such benefits as being able to eliminate media server sprawl as well as faster backups and restores.


Although architecture and related functions, especially automation, are general themes for Dell EMC data protection for the SDDC, the company also made specific announcements at VMworld 2017.


The first focuses on data protection for virtualized mission critical applications. This utilizes the Dell EMC Data Protection Suite for Applications applied to a VMware-based environment to ensure the necessary levels of data protection. In order for an IT organization to move a mission-critical application to a virtualized SDDC environment, the application must receive the same enterprise-class service-level agreement support (such as uptime availability and fast application restoration if a downtime problem were to occur) as the application received in its legacy environment.


As an example, the application suite member delivers self-service control to Oracle and SQL Server database application owners for day-to-day activities. That ensures that the overall environment is under IT governance and control to make certain that application data protection compliance, availability and performance (such as time to recover) remains paramount with automation being key to the process.


The second announcement is a new Dell EMC data protection solution for hybrid cloud environments leveraging VMware Cloud on Amazon Web Services (AWS) where the goal is cost containment (on Amazon S3) or overall performance (with Amazon EBS). This hybrid cloud solution integrates on-premises IT infrastructures to enhance data protection of AWS-based assets. It also features automated operations both on- and off-premises, as well as what Dell EMC claims as best-in-class data deduplication.


Mesabi musings


Dell EMC has never rested on its data protection laurels, and is unlikely to ever do so. The company has rightly recognized how many of its enterprise customers are making a concerted move to VMware-based software-defined data center environments. As a consequence, Dell EMC is proactively evolving its data protection portfolio to address the needs of the SDDC and those customers.


A major architectural focus in this transition has been on automating the software that manages all levels of the data protection stack, from policy management at the top to the data protection storage itself at the bottom. Specifically, Dell EMC has enhanced the Data Protection Suite for Applications family member to deal with SDDC-supported mission critical applications. It has also enabled data protection support for VMware Cloud on Amazon AWS. By doing so, Dell EMC is continuing to build on its already important leadership role in data protection.

IBM Thrives in the Storage Patch

IBM’s public announcement that external storage systems revenue grew by 7% in Q1 2017 and 8% in Q2 2017 reverses an ongoing downward trend. Now, past performance is no predictor of future performance as famous financial disclaimers always state. However, this is excellent news for IBM, especially since it does not seem to be a case where the tide raised all the storage ships, but rather one in which the company appears to be a positive exception.

That is also good news for IBM’s storage customers since the company should continue what has been a successful direction in its storage R&D. Thus, customers should feel more comfortable with both their current IBM storage investments and continuing future storage innovation.

That may seem to be a bubbly interpretation of just 2 quarters of positive revenue growth, but it comes on the heels of confirmation of IBM’s leadership status in software-defined storage (SDS), which is one of the major driving trends in the storage market (see ). This upward SDS trend has continued for some time. There does not appear to be any reasons why IBM cannot maintain a successful leadership role in SDS. In fact, there are a number of reasons why IBM should continue its successful run.

Tectonic changes enable IBM to play on all server platforms

All storage systems work in conjunction with servers. A quarter century ago, then direct-attached storage (DAS) was dominant, server and storage vendors were the same. Then EMC came along and started to dominate the mainframe storage market in the era of shared storage and Fibre Channel (FC) networking.

The result was that server and storage customer-facing companies (namely Dell, HP, IBM, and Sun) had to focus on trying to protect storage sales that connected to their own brands of server. Thus, in general, IBM did not sell storage to HP server customers and vice versa. However, storage-focused companies (EMC, as well as Hitachi Data Systems (HDS), and NetApp) prospered by selling their storage solutions in conjunction with any and all server platforms.

Companies with a perceived server heritage, such as IBM, were put on the defensive. But that model is no longer valid. As a result of significant acquisitions and internal investments, IBM is now on the offensive in selling storage across all server platforms, as well as recapturing business on its own server platforms.

How did that come to pass? The old model eroded in the last decade due to ongoing major trends in the information infrastructure as a whole and storage in particular. First, the vast majority of non-IBM POWER servers are Intel-based, and the focus is on operating systems from Microsoft and open source vendors, like RedHat, as well as server virtualization, such as from VMware. IBM has worked very hard to provide a broad and robust range of storage solutions suited for these environments.

But the key is that IBM is positioned to take advantage of broader trends in the information infrastructure that parallel trends in storage. Enterprises want to move forward with both their traditional applications that have long sustained their businesses (current processes) as well as new generation application workloads to transform the business (such as cognitive era processes to gather and apply insights to generate new sources of revenue).

Traditional applications are efficiency-driven with high levels of service delivery being a key objective. All flash storage, such as provided by IBM, is one way to achieve this objective while preserving legacy software. (See On the other hand, new generation applications and workloads, like big data analytics, can take advantage of the new software-defined infrastructure alternatives, where IBM offers a variety of SDS choices. (As before, see ).

A corollary to the old adage that “No one ever got fired for buying IBM,” was that no one who did not have IBM servers already would consider buying storage from IBM. In sharp contrast, today customers will consider IBM as a potential storage alternative whether or not they have done business with IBM before.

Regaining the mainframe storage patch

Of course, IBM wants to take advantage of working with new customers who have never previously used its storage solutions, but that does not mean that it ignores its own storage patch. A key example is the effort that IBM has been making to recapture and strengthen its position in Z Systems mainframe storage. One of the ways that it has done so is through the use of all-flash for its DS8880F products. (See ).

In addition, the company has also offered innovations, like Transparent Cloud Tiering, with the DS8880 products, which takes advantage of key trends in hybrid cloud and cannot be easily duplicated by competitors. (See ). IBM’s July announcements included enhanced storage performance, such as up to 50% cut in response time, and increased capacity: up to 2PB in a single enclosure, which is a 3.8X improvement over past-generation storage systems.

Mesabi musings

The storage market continues to grow rapidly in terms of volume, but improvements in price per volume of storage and the diversity of competitors have made it challenging for established vendors. IBM appears to have more than risen to those challenges and seems to be thriving.

One reason is the company’s success with delivering storage innovations for established customers, notably the DS8880 storage for Z Systems mainframes. But it also seems that IBM now appeals to all storage customers and not just the ones who have traditionally bought its servers. This is a good thing for customers since they now have access to a broad and diverse range of IBM hardware and software options, and an energized company bent on delivering future innovations. Customers are provided more choices in the storage decision-making process, and that also increases the likelihood that they will stick with IBM.