Tuesday, March 29, 2011

zEnterprise - your next process control computer?

Some may recall that during the nineteenseventies and eighties mainframe computers entered the control rooms of some Northamerican refineries. I recall a visit to Imperial Oils Strathcona Refinery located in Refinery Row while I was studying in the computer process control group at the University of Alberta. We were showed around in the plant and then control room beforea Q&A session, and we were quite amused that the control screens - green text on black background - showed temperatures with four digits after the decimal point. Those were the early days of virtualization on the mainframe. In those days mainframe computers were only available with one propriotary operating system and I seem to recall green characterbased terminal access. Of course this fitted well with the character based lineprinters available at the time.

However, since those days things have changed. I have seen process simulators requiring more than 15 of the most powerfull multicore x86 based personal computers to give a reasonable response to operator console inputs. If you wanted to run the same in simulation mode, then add another 15 computers plus the necessary cabling, routers, network cards etc. All potential points of failure. Properly the company with this simulator also had personal computers for other parts of the business, e.g. process control, maintenance, and of course their business processes. Thus is is not uncommon for a single site to have a significant number of x86 based servers at each plant location. These servers of course require cabling, power, maintenance, updates etc. Maintenance and updtates are potential outtages. What is the MTBF of a single x86 based server? Some say two months if only considering hardware. Then what is the MTBF of our networked system of more than 15 x86 servers?

Now consider the newest mainframe computer zEnterprise 196. A couple of days ago I had the opportunity to attend a siminar titled "Smarter Linux for Enterprise Systems" at IBM Denmark. At this event features of the zEnterprise were demonstrated in live demos and technical presentations plus of course a business presentation about why these boxes would benefit your organisation. The zEnterprise has a MTBF of 30 years (Measure by having 30 systems in one location for a year). The zEnterprise will run hundreds or thousands of instances of the latest linux from either RedHat (RedHat Linux Enterprise Server 6.0) or Novell (SUSE Linux Enterprise Server 11.0) - of course with graphical interface from your laptop or other personal computer. It also comes with mainframe level security - a mainframe is still to be hacked, and it has been around for more than 40 years!

A mainframe system comes with a significant starting cost, but consider: You can run your process control software on one linux instance right on the hardware, and chose to run others systems, such as your similator and maintences systems on top of the hardware virtualization.

I you want further segregation of systems, then you may add one or more blade center extensions, which may run either linux based blade centers or x86 based blade centers. So you could have one x86 based blade center running your current Windows based control software, with another x86 blade center as a hot standby. This is just like in the old Honeywell TDC 2000 control systems, where one box served at a hot standby for seven other boxes on the same control net.

Consider that you get all this without exensive cabling between the servers, and with dedicated high speed link between the zEnterprise 196 and each blade center extension. Even disaster recovery comes cheap. If you need another zEnterprise for disaster recovery, then you only pay a fraction of its list price to have it available at your site.

Beside cost savings due to reduced cabling, increased security in face treaths such as stuxnet, better disaster recovery and testing, there could also be licence cost savings especially if you use commercial x86 databases at your site.

A further benefit is, that all these systems can interface to the same storage system. This allow you to easily give engineers, business analyst and others access to historical process control data on your mainframe based business network.

So next time you are shopping for a process control computer, and a simulator, and a maintenance system, then take a look at the newest mainframe. You may like what you see, and to run it you just need people with linux skils.

NOTE: I have never worked for IBM or any other mainframe company. My only connection with IBM was a user of the IBM 1800 Process Control Computer in the Computer Control Group at the Department of Chemical Engineering at the University of Alberta in the late seventees.

Thursday, March 03, 2011

Follow-up on an accident

During a lunch break an employee left a shovel near a restaurant. Another employee picked up the vehicle and went for a ride. Along the ride this employee picked up to more employees. The ride ended with an accident just outside the site. In the accident all three employees were seriously injured.
Questions that come up are
1. Why did the employee pick-up the vehicle?
2. Why did the employee pick-up other employees for the ride?
3. Is the accident a recordable incident? Does it affect LTI?
4. Should the employees be disciplined? How?
5. Would the workers be eligible for workers compensation for their injuries?
6. Should the company investigate the incident? Or leave it to the police, since it happened on public property?
Generally people working with safety for a company has at least two responsibilities
To ensure that company employees has a safe workplace, and goes home each day as safe as they arrived for work.
To protect the company from lawsuits by ensuring, that the company has the required procedures in place to ensure the employees are capable of performing their duties at work.
These dual responsibilities can of course lead to conflicts.

Accident investigation
The first two questions are clearly part of the accident investigation.
One need to determine the first employees intent when picking up the vehicle, and also his/her reason for picking up the other two workers. Also statements from vitnesses seeing the vehicle on its route from the resturant parking to the place of the accident would be relevant to the investigation of the event. Notice, that this would normally fall outside a police investigation of the accident, since that would only involve immediate causes.
Also outside the police investigation lie questions about whether company procedures for using company assets were followed prior to the accident.

Statistical impact
The third question would clearly depend on the jurisdiction, although one would expect some similarities among Western European countries – especially the EU, or between provinces in Canada, or states in USA. Although appearantly within Australia there are difference among the different states. Many would immediately look for a definition of reportable events in applicable legislation, but many times this is not sufficient. Some will look for definition in standards such as OHSAS 18001.

When working in Sarnia in southern Ontario in Canada many years ago I and a colleague had a vihicle at our disposal for transportation between the site main office and the plant. Since the main office was not located conveniently for public transport we often used this vehicle as part of our transport to and from work. During the night the vehicle was parked on company parking lot which was located conveniently for public transport. Was this use authorized by our manager? Definitely not! We also often used the vehicle to drive to a nearby donut store during lunch. Was this use authorized? Definitely not! Did we ever consider what would happen if we had an accident during these uses of the vehicle? No, it never entered our radar screen. Many other employees used company trucks to drive to the same donut store during lunch time, so at least that was at the time considered acceptable.

Some argue, that the incident is not recordable because it is not site related. However, a few years ago the annual report of Shell UK feature a report of two fatalities – both to subcontractors hired by logistics part of the company, and both occuring far from the site.
At the time I was working at a plant in Sarnia a down stream plant had an unfortunate release of perchloroethylene – a dry cleaning solvent in August 1985. The material quickly settled on the bottom of the St.Clair River, the release was immediately reported to relevant authorities. Somehow Greenpeace got word of the release, and in stead of the companys information about the release and the dangers of the substance released media across Ontario picked up Greenpeace's version of the story about the release. Despite company and authority efforts to analyse the content of the blob and release this information to the media even 20 years later it is Greenpeae's version which the media cites (see e.g. “Canadian petrochemical plants blamed for gender imbalance” by Paul Webster in The Lancet, Volume 367, Issue 9509, Pages 462-463, 11 February 2006, and available on the web-site www.lancet.com ). This shows the importance of not just doing what is legally correct, but that you must also be seen to do the correct thing in the eyes of the public, i.e. the media.

Some even ague, that the recordability of the events depends on whether the employees where payed during their lunch break or not.

The bottom line is, that it is up to the company to define what should be reported and how it should be reported when company employees and/or company assets are involved in an accident. In companies operation multiple site in different jurisdiction company procedures and standards w.r.t. reporting accidents become more relevant, since differences in the legal requirements may make company reporting, e.g. in annual reports to shareholders more difficult.

Penalty impact
Some argue, that the employees involved in accident should be penalized, but that firing them is properly going too far. I am not sure that a penalty will have any positive effect. However, a prudent company would at least record the event on the employee file.

I was once told about a colleague of mine, who had difficulty leaving the liquor bottle alone, and often showed up for work under the influence of alcohol. This employee had been on several courses to get rid of the drinking habit, but it did not seem to help. Finally his supervisor told him, that if showed up for work one more time under influence of alcohol he would be immediately fired. He stopped drinking from that day. Unfortunately he started drinking again right after going on retirement, and within relatively short time managed to drink too much. In his case the treat of a penalty worked as long as the incentive to keep his job was there. After the incentive disappeared the old habits quickly came back.

Compensation impact
The fifth question also would clearly depend on the jurisdiction.
Some argue, that the workers are not eligible for compensation, since they were not performing a work activity. However, at least the first employee could argue, that he/she was just bringing the vehicle back to the site when the unfortunate accident happened, and he should have workers compensation because he/she was bringing an abandoned vehicle back to the company site. The other employees could argue, that they were just attempting to get back to the site faster.
Other correctly points out, that the question about compensation could involve determining whether or not the employees where payed during their lunch break.

Clearly the company is responsible for company assets and the proper use of these assets. If the company has neglected this responsibility by not having appropriate procedures in place to ensure adequate training, then the company would be at risk in some jurisdiction, e.g. in Denmark, of being judged as having failed to take every precaution for the safety of the worker, as the legal language in some jurisdiction define the responsiblity of the employer to provide a safe workplace.

Learning impact
The sixth question would depend on the safety culture at the company.
A company with a well developed safety culture would properly investigate the incident to learn as much from it as possible, so similar event could be prevented in the future. But the event is definitely an opportunity for learning.

If the company has procedures in place for training employees or otherways ensuring employees are training to properly handle company vehicles, which they need to use during their work, then it becomes much easier when an accident happened to decide what to do as a follow up on the event. The lag of such procedures could lead to an implied policy, i.e. in my use of a company car during part of transportation to / from work.

Conclusion
So what can one conclude from the above? One conclusion is, that any accident involving employees and / or company assets should be investigated. Simply because there is an opportunity to learn, and prevent similar events in the future.

My former employer in Sarnia, Ontario actually kept track of injuries sustained by employees in their spare time, that prevented them from comming to work the next day. Based on that information departmental safety meetings were arranged to to cover topics such as proper warm-up prior to exercises such as squash or handball games.

The thoroughness of the investigation should depend on the learning opportunity for the organisation as a whole. Someone need to make that call, and I think it should be the local safety manager.

What do you think about off-site accidents such as this one? Would you take the opportunity to learn, or would you rule it out as not relevant for your business?