Skip to main content

genchi genbutsu


Often, I see technicians dealing with issues and problems while sitting firmly on the seat of their pants at their desks. Call me old school, but I do not think this works! There is no substitute for the eyeball.

When I was a technical account manager at Madge Networks, I received a call about a problematic branch of Standard Bank. It was their flagship branch in Sandton. No expense had been spared and it was clad top to toe in marble. However, the staff at branch regularly reported that the network became toast. As is token-ring's self-healing nature it would correct itself automatically. The temporary outages were causing a great deal of frustration and technicians at the bank had been unable to diagnose the fault. I decided to investigate. On the first visit that I was at the branch I plugged in my notebook and did a health check of the network. It was running as smooth as a Husqvarna sewing machine. My mom owned one of those from the late sixties. I left and resolved to come back later.

On my second visit, I was sitting at my notebook in the room where the MAUs (token-ring hubs) were racked sifting through network traces looking for the proverbial needle in the haystack. The room where the network equipment was racked also served as the stationary store and had some mainframe printers. My thoughts were disturbed by one of the mainframe printers kicking into life. The print job completed and a few minutes later a staff member arrived to retrieve the print job. She walked over to the printer and the next moment the hubs started "shotgunning". "Shotgunning" is the distinctive sound a token-ring network makes as the relays open and close when a fault occurs. The self-healing nature of the network kicks in as it starts attempting to isolate the fault. The manager of the branch ran into the room and excitedly asked if I had seen and diagnosed the fault. I assured him that I had captured the fault on my equipment and if he could give me a moment to analyse it, I would provide some feedback. But first, I walked over to the printer. It was located at the end of the room and a passage was left to reach it between piles of stationary boxes. Halfway down this passage was a piece of carpet covering a token-ring fly lead. Suspiciously I lifted the carpet, and immediately eyeballed the problem.

The human traffic over the carpet to the printer had badly frayed the fly lead and each time someone went to fetch a print job the frayed fly lead would short and cause a fault. We replaced the fly lead and moved the printer to a more suitable location and the problem was solved. Without being present and doing visual inspection many problems will remain unsolved. Thus, the most important aspect of solving problems is to go and have a look.

The Japanese are extremely familiar with the concept of go and have a look and to them it is known as genchi genbutsu. At its essence in means you need to develop, and design products based on being at the coalface and seeing the problems, not hearing about them second-hand. You need your own eyeballs and not a report given to you by an intermediary. Only then do you provide added value and solve problems. Even though this Japanese concept has been with me throughout my working life, I am not Japanese.

One morning before I arrived at Investec I had a phone call on my mobile that the dealing room wasn't able to operate. Connectivity was lost to the main server and the network was being blamed. I arrived at the dealing room and was accused of network changes the night before. Highly unlikely, I quipped, I was at a company event were buckets of beer were on offer and I was not capable of changing anything. So, I asked them to show me the problem. The application wasn't able to connect to the server but from a DOS box we could ping the server. The server IP address was 10.0.32.99

After a few minutes of head scratching I loaded Ethereal, now known as Wireshark, a packet capturing tool. One packet capture later I worked out that:

  • the app did a DNS lookup first before using the server name field as an IP lookup
  • the app was configured with 10.0.32.99 The IP was not assigned an A record in DNS.
  • the app did a DNS lookup on 10.0.32.99.investec.co.za (this failed), then it tried 10.0.32.99.co.za, and a result was returned for an A record out on the Internet,

It seems that overnight someone in South Africa registered the domain 99.co.za and as soon as that happened things went pear shaped. I resolved the problem by creating a local hosts file with a server name for 10.0.32.99 and configuring the app to use that server name.

After all this I expected the IT manager of the dealing room to appreciate the effort like the dealing room manager in London. How wrong could I have been! She said how dare someone register 99.co.za without asking her permission and why did I allow it. I was gob smacked for a while and then I let rip. It boggles the mind that people are appointed to manage Information Technology but do not understand the basics! My manager at the time called me in and said she did not like the way I talked to people. I was young and naive.   In the world more and more people operate outside the sphere of their abilities. Even a real estate developer can become leader of the free world. In South Africa it is common for cadres of the ruling party to be deployed as Ministers in fields to which they have no clue about the fundamentals in the area in which they are the leaders. This has been the case in telecommunications which is a to put it mildly, a rotting roadkill golden wildebeest carcass .

I was working at an ISP where the neighbours were building a multi-story building next door. One morning driving in a saw a car in the visitor's parking lot covered in cement. I was initially flabbergasted that such an incident could happen but when looking up from that parking lot, it was obvious that it was a matter of time before something like that happened. The path of the crane arm was directly overhead. Often when you have a look, the potential for an incident is there. If you don't look closely or you in a hurry you end up with your car covered in cement.

A bolt fell from the roof of a vendor neutral data centre. It smashed the windscreen of a car. A few minutes before that a number of people including myself had walked past that point which was a few metres from the entrance. Having car parks in areas that are dangerous seems to be common.

One of the companies I worked at has a set of values: distinctive performance, dedicated partnership, client focus and cast-iron integrity. These are laudable values, but you do not need to work at a specific company to live these values or any other values. In many cases people within the company do not even live the stated values.

Sir Richard Branson summarized all business values in a single sentence which is that it is all about doing the right thing. Most companies do not realize that the priority in life for individuals lies elsewhere. A company can easily replace an individual, but that same individual cannot with the same ease replace his family.

The difference between a good company and a great company is that a great company will always give more than what it takes. To this end the priorities of the individuals need to have the same level of focus as that of the company’s customers. The reason is simple in that without a happy employee, the company has a diddly squat chance of having a happy customer.

I worked at an ISP where the boss would regularly force his staff to compete against each other and in meetings he would kick the walls. It wasn't an environment where you excelled, you just existed.

Comments

Popular posts from this blog

Why Madge Networks, the token-ring company, went titsup

There I was shooting the breeze with an old mate. The conversation turned to why Madge Networks which I wrote about here went titsup. My analysis is that Madge Networks had a solution and decided to go out and find a problem. They deferred to more incorrect strategic technology choices. The truth of the matter is that when something goes titsup, its not because of one reason only, but a myriad of them all contributing to the negative consequence. There are the immediate or visual ones, which are underpinned by intermediate ones and finally after digging right down, there are the root causes. There is never a singular root cause for anything but I'll present my opinion and encourage everyone else to chip in. All of them together are more likely the reason the company went titsup. As far as technology brainfarts go there is no better example than Kodak . They invented the digital camera that killed them. However, they were so focused on milking people in their leg

Flawed "ITIL aligned"​ Incident Management

Many "ITIL aligned" service desk tools have flawed incident management. The reason is that incidents are logged with a time association and some related fields to type in some gobbledygook. The expanded incident life cycle is not enforced and as a result trending and problem management is not possible. Here is a fictitious log of an incident at PFS, a financial services company, which uses CGTSD, an “ITIL-aligned” service desk tool. Here is the log of an incident record from this system: Monday, 12 August: 09:03am (Bob, the service desk guy): Alice (customer in retail banking) phoned in. Logged an issue. Unable to assist over the phone (there goes our FCR), will escalate to second line. 09:04am (Bob, the service desk guy): Escalate the incident to Charles in second line support. 09:05am (Charles, technical support): Open incident. 09:05am (Charles, technical support): Delayed incident by 1 day. Tuesday, 13 August: 10:11am (Charles, technical support): Phoned Alice.

The best social media requires no batteries

  Today it is all about social media such as whatsapp, facebook, twitter or even LinkedIn. However, the best social media is Craic. No, it is not to be confused with substance abuse. Let me explain. Often when people meet around a braai , dinner table, or share either a pot of beer, bottle of wine, a cup of tea or a mug of coffee a conversation is likely to happen. This conversation is invariably about things and is referred to as Craic. And it is best reinforced with a good bottle of whisky (typically an Irish one, which would be known as a whiskey). Now talking about why some people call it whisky, and other whiskey is good Craic. Craic is often a discussion about things that spark a debate or lead to an extensive and prolonged engagement. Things in our world are objects that exist or have existed for a long time period. We typically assume that things in our modern world have only been around a short time period but invariably many