Cisco TAC Engineers - Research Proposal Example

Summary: Cisco TAC engineers rely on their knowledge of the technology they support, escalation engineers, internal tools like topic search, and Cisco documentation from the internal website. TAC engineers, especially those with less than 2 years of experience, spend most of the day researching issues and trying to find a previous case on the topic tool to find an answer or a resolution to the ticket at hand. In many cases, the topic tool is worth the time spent researching, and the engineer usually is able to find a tip on how to solve the case, and, in some rare cases when the previous ticket is nicely documented, the engineer is even able to find troubleshooting commands as well as specifics on how to solve the customer’s issue. Escalation engineers are usually the best resort for the TAC engineer but, given the ratio of escalation persons to TAC engineers, a TAC engineer will ONLY reach out to his escalations when he doesn’t know the answer, or is unable to locate a Cisco document that would solve his issue and is not able to find a similar previous ticket or at least a previous ticket with valuable and relevant information. Topic tool, however, is at the TAC engineers’ disposal at any time and readily available with a tremendous amount of information. The real challenge is being able to search for the information you need and be able to find the answers you want on topic in the shortest time possible. The goal is not only to solve a case but to solve it quickly. Experience has shown that the more information added to the topic database and the shorter the time required to find that information equals the shorter the resolution time. This enhanced the overall customer’s experience with TAC. In some cases, the customer is a person with more experience than the TAC engineer himself but they still call in with a firm belief that their issue will be resolved because they know that a TAC engineer has the resources and will be able to find the answer from a previous ticket, filed notice, or a documented bug, all of which reside on our topic database. Out of all the resources available for TAC engineers there is consensus among TAC engineers that they get most of the answers to resolve their cases from the information found on the topic tool. The goal of this proposal is to make topic search a richer tool by adding more information to it, hence enabling TAC engineers to find solutions to a wider spectrum of issues. Based on my investigation, research and discussions with other engineers I found that an engineer is able to find resolutions for about 70% of his cases from the topic database. One of the goals of this proposal is to allow a TAC engineer to be able to resolve 90-100% of his cases based on the information on the topic database. Based on the experience of many TAC engineers, another issue is not being able to find the desired answer in a very short time. Sometimes hours of research and digging on the topic database are spent before you arrive at the case that would help you solve a certain issue. Typically a TAC engineer types in a brief summary of the symptoms of the issue and is presented by a huge amount of cases that will possibly have the correct answer to his query. The cases are presented and ordered based on certain factors that reflect the topic tools opinion on which cases are probably more valuable and will provide the desired answer to the engineer. The case with the highest potential is listed first and the second is listed second and so on. Realistically, the search engine’s opinion may not be exactly the most accurate and the most relevant previous case may be presented as the second or even on the third page in the search results. In that case, a lot of time is spent by the TAC engineer skimming through a huge number of tickets before finding the desired case that would provide the most helpful notes, commands, links and more. This proposal also suggests adding fields to the ticket structure enabling a TAC engineer to weigh in and give his opinion on which case he thinks is very helpful on a specific topic. This can be accomplished if the TAC engineer can fill out certain fields before closing the case, marking that case as a good resource on a specific topic, sub-technology or as a network protocol. To reduce the amount of time needed to find an answer in the topic tool, this proposal suggests an enhancement of adding new fields to the ticket structure. This will enable the TAC engineer to find the answer in a quicker manner. With the new fields added and properly populated by case owners, the TAC engineer will be able to find the answer he is looking for on the first case or two presented on the search results versus having to skim through many cases before finding the most relevant case on the second or third page of the search results. Today, the engineer queries certain terms on the topic search bar and the tool relies on search intelligence or Google search intelligence to present the engineer with the best results according to the search engine’s opinion. The suggestion is to add human intelligence to the decision process since the TAC engineer is the user of the tool and not an average person using a random search engine to search through the internet. We would like TAC engineers to be part of the decision of which case is most relevant to a query run by another engineer. This will add to the relevancy of the cases presented. This proposal will not require us to ask our search tool vendor “Google” to make changes to their algorithm. We simply need to make a few additional fields to our ticketing system which can be accomplished at an extremely low cost, if any cost at all. We also need to encourage TAC engineers to change their behavior slightly when updating and closing cases. This proposal will make it almost effortless and timeless to find an answer to a specific problem on the topic database in the future. It would be extremely helpful to assist a TAC engineer to find an answer to his query or question on the first two or three cases he reviews or within the first 10 minutes of his search commencement. On average, today a TAC engineer spends about 1.5 to 5 hours searching on the topic tool to find an answer/resolution to a ticket at hand. Since the goal is to enrich our knowledge database, the results of this proposal can be measured in about 3-5 months after the changes to the ticketing system are incorporated and that TAC engineers successfully changed their behavior on documenting closing cases. This period of 3-5 months will be enough to have several rich quality cases with valuable information on each specific topic or issue type. This proposal will create a more operationally efficient system and leverage our human capital to what we do best, and that is customer support. TAC engineers are highly talented hardworking people and through the better use of resources suggests that utilizing the engineer’s knowledge in building a better database that will help future TAC engineers solve problems faster and reduce resolution time dramatically. Implementing this solution will reduce ticket resolution time, which can ultimately reduce the TAC operational costs and enhances the customer’s experience. Problem summary: A recent examination revealed that the topic database does not always provide engineers with the answers that they need. Another observation is that the time to find an answer “if one exists” is not short and, in some cases, enormous. Objective: Given the importance of building a very rich topic database with valuable information that would enable TAC engineers to find answers easily and quickly regardless of their experience level, and given the importance of mining that database and finding the right answer and presenting it to the customer in a very short time, it is unlikely that demonstrable results will be likely in the first 3 months. The worth of our topic database is measured by the amount of content and information we have saved and that translated to well documented and structured tickets, and adding content takes time. During the first month, emphasis will be placed on setting up the project, making changes to the ticketing system and encouraging TAC engineers to contribute to the growth of the database and be generous when updating cases with their knowledge. Implementing this solution will reduce ticket resolution time which will ultimately reduce the TAC operational costs. To assess the extent to which the proposal remedies the two problems noted above it would be necessary to quantify the results to measure the projects effectiveness. Below are the recommendations: A/ To reduce the time spent by TAC engineers in finding a resolution to a problem from searching the topic database: The Google algorithm has been closely examined since the tool discussed here is a Cisco tool offered by Google. Topic is a Google tools and uses an algorithm similar to Google search algorithm to find results on the world wide web. In general, Google takes a couple of factors into consideration before they decide which is the best result for a certain query, whether its their globally-known search engine or the simple engine provided to Cisco known as topic. Google looks at a website’s title and matches that against the user’s query. They also look at how many times the user’s query or the word searched by the user is found on a given website (in our case, a ticket, not a website). This is known as keyword density. If a user is searching for the word “car”, a website that displays the word car 5 times has a higher chance of showing up on the first page than a site with the word “car” mentioned only 2 times. Another factor is inbound links. This is how many websites (in our case, tickets) have a link to a particular website (ticket). A link from another website is like a vote to your site. If many websites out there have a link with your site url or address, the more trustworthy you are in Google’s eyes and the higher you will rank on search results. The same applies to Google’s topic search. The more links a ticket has from other tickets, the more trust it gets from the search engine, increasing the likelihood of the ticket showing on the first few results. This link metric will be discussed as one of the following objectives. The most important factor in Google’s eyes is the user’s opinion. They measure how much time a user spends on a site and they take that into consideration. This is considered to be the most important metric out of all the ones discussed above because it gives a true sense of the importance of the material to the users. If a user is spending 20 minutes on a site as opposed to 3 minutes on another site, this is a clear sign that the 20 minute site has more value from the user’s perspective and answers their questions or provides the information they are looking for. There is consensus in the internet community that this is the most important factor, as well as the smartest Google factor because they are not relying on engine intelligence but allowing the user’s experience to be part of the decision. They are relying on data gathered from real users and Google is gauging their interest on a site. By the same token, Google asks web masters to make sure to put in the words that better describes their website’s theme and preferably if there words are mentioned several times on a page. This will also help the engine decide what is an authoritative site on a given topic based on the keywords mentioned on the site and their density. By the same token, we can use the TAC engineers’ input and take that into account before the topic search engine decides which case is best and need to be presented first. Once an engineer is ready to close a case he should be presented with a simple form or field asking him to put 5 keywords that he thinks best describes the issue or the case topic. The engineer will then add 5 words that are a lot more descriptive to the nature of the problem that the description of the problem added to the ticket by the customer or the CIN agent. The 5 keywords will become part of the ticket notes and will make this particular case surface on the front page or the results for another engineer in the future who is searching for any of these 5 keywords. The TAC engineers’ ability to add 5 keywords will be very helpful even if the 5 keywords already exist on different places on the case notes, as it will increase the keyword density on a given case. Keyword density is the percentage of times a keyword or phrase appears on a ticket compared to the total number of words on the ticket. In the context of search engine optimization, keyword density can be used as a factor in determining whether a ticket is relevant to a specified keyword or keyword phrase. Adding the keyword, for example, "high cpu" by a TAC engineer to a case that already is addressing a high cpu issue and has the word high cpu mentioned on the case notes once or twice will make this case a potentially top result for another TAC engineer searching for the words “high cpu”, which is a desirable goal. In cases where the keyword was not even mentioned once on the case notes that these additional 5 keyword field is extremely important, as it will mark the case as a potentially helpful case in a specific topic and cause it to show up on the results. Without the keyword field, the case would not be visible in the topic since the ticket body did not have the correct word. B/ To encourage TAC engineers and senior engineers to put in more information on tickets increasing the overall value of our ticketing system as a data mine: After reviewing more than 40 cases, I noticed that all of the tickets have minimal to average content. The TAC engineer identifies the problem, sometimes mentioning the steps taken in identifying the issue, and makes a general statement about his resolution; in most cases, the engineer does not discuss the specifics of the resolution on the case notes. What I mean by specifics is how the problem was diagnosed, what led the engineer down the right path of identifying the issue, what commands he used to determine what the problem was, and what set of commands he used to fix the problem. This may seem like too much information to add, but given the time spent working with the customer and identifying the issue, the time needed to perfectly update the case notes is relatively small. If all of this information is added, another TAC engineer will know exactly how to solve a similar or related problem in a very short window since he has all the information on the previous case down to the command level. This will be very helpful in driving down resolution time as opposed to a new engineer only finding a brief problem description and a small reference to what the answer was. It will be a breakthrough if we can convince the TAC engineers to share all the knowledge utilized on a given case with other engineers potentially reviewing the case in the future. Different people have different approaches to things, different personalities and behaviors. Some may feel they already resolved the ticket and don’t realize the benefit of sharing the specifics or the commands they used and are okay with a simple description to utilize their time doing other things. Others may not be very comfortable sharing a great deal of tricks or commands with others, while others may just not be so patient when it comes to detailed documentation. However, it is rather easy to find a mechanism to encourage all engineers to share as much as they can to increase the amount of information and the worth of our ticketing database. One approach would be to add additional fields to be filled out by engineers prior to closing a case. One field can prompt the engineer to enter the commands he used to identify and rectify the problem. Another field for the engineer is to list any documentation from the Cisco website that he used to identify or fix the issue. Three fields of “commands to identify” “commands to fix” “links to documents used” will provide a very rich ticket structure with a lot of valuable information. Another approach would be to add a rating field to each ticket, allowing engineers who come across a case to give the ticket a rating on a 1-10 scale. A case that was properly documented with all of the technical details and the commands or is very helpful to the engineer performing the search will be given a 10, where a poorly documented case will be given a 1. Many TAC engineers are keen about building a reputation inside and outside TAC with other groups. This rating mechanism will encourage engineers to contribute more when documenting their cases, which will help them with their personal goals if they maintain a good rating on most of their tickets. C/ Changing a case title which was not insightfully written by a customer Many of the case titles are not insightful and do not explain the nature of the problem at hand. A customer usually writes the case title based on his perception about the problem. Case 609231579, for example, has the title of "AP issue". According to the customer, this is the best description to the problem. In reality, this title is not very helpful since future engineers will not be querying the topic search tool with the phrase "AP Issue". If the owner of case 609231579 modified the case title to "AP unable to boot/flash corrupt" this will be helpful. Other TAC engineers having a flash corruption issue will easily be able to find that case when they query with the keywords "flash corruption" or "boot issues". The more matches we have on the case title and the case body to the searched keyword, the higher the chance a case will show up on the results. Therefore, modifying the case title will highly increase the likelihood of the case being found by future engineers experiencing a similar or related problem. A good analogy, Google to internet websites is like a topic search to the ticket database we have. The same way Google searches websites and present us with the best results, the topic tool also searches all tickets and presents us with the best, most relevant tickets. In this analogy, the case title is the website title or name. Naming your site that sells cars “cars.com” will definitely attract more car searching users than another site called “abc.com”, which also sells cars. Having the appropriate case title gives the Google algorithm, whether the one used on their search engine or the one incorporated on our topic tool, an idea that this is an authority site on the given subject, much in the same way our case titles will mark our cases and tickets as authority tickets on certain subjects. There is no cost in changing or correcting a case title, as an engineer simply can go into workbench and change the title to a more descriptive one. We just need to make that a habit. Once the TAC engineers are already updating the case title with the most relevant keywords then other engineers can easily find the answers they want. Instead of performing general queries against the entire ticket, we might be able to use specific commands to search for keywords only on case titles. For example, putting the phrase "allintitle: AP.issue" on the topic tool will only display the tickets that have the words “ap” and “issue” in the title. These cases will most likely be the ones an engineer is mostly interested in because the title was updated by a TAC engineer and not by a customer. We can achieve that level of granularity on our search once we get a confidence level that our case titles are precisely descriptive to the problem and are added by our engineers. These granular commands have no meaning and no value today because we can not rely on case titles added by customers. There are different variations of the intitle command that can be used on our topic search and these can be introduced to TAC engineers once we already have our correct titles and everything else is in place. D/ Linking a case to another case Linking a case to another creates a case analysis algorithm used by topic search to assign a numerical weight to each element “case” of a set of cases linked together with the purpose of measuring its relative importance within the set of cases. Studies have shown that this is a very productive approach and helps to determine the most valuable component in a set “ticket in a ticketing system”. The model has been developed from a Stanford University patent "Method for node ranking in a linked database Patent (U.S. Patent 6,285,999)". This algorithm relies on the uniquely democratic nature of the ticket database by using its vast link structure as an indicator of an individual tickets value. In essence, topic interprets a link from ticket A to ticket B as a vote, by ticket A, for ticket B. However, topic looks at more than the sheer volume of votes, or links a ticket receives; it also analyzes the ticket that casts the vote. Votes cast by tickets that are themselves "important" weigh more heavily and help to make other tickets "important". In other words, a weight results from a "ballot" among all the other tickets on the database about how important a ticket is. A link to a ticket counts as a vote of support. The weight of a ticket is defined recursively and depends on the number and weight metric of all tickets that link to it ("incoming links"). A ticket that is linked by many tickets with high weight receives a high rank itself. If there are no links to a ticket then there is no support for that ticket. This is already built in our ticketing system, hence will not incur any additional cost. The TAC engineers are just not aware of the importance of this linking concept and we need to bring awareness and encourage TAC engineers to link their current cases to the previous cases that have helped them the most to come to a resolution. They will help the topic tool to get votes on all previous cases stored and assign values to different cases increasing the likelihood of topic giving us the case we need. A minority of TAC engineers today are linking cases to bugs or to a case that is a bug case. We want to take this to another level and link every single case to the case that was the most helpful to us. E/ Adding a link to the CCO documents used Many TAC engineers also rely on the CCO tool for steps to accomplish a certain task or resolve a certain issue. Documentation is another source of information heavily relied on in TAC. It would be great if we added a field to our ticket system with the name "CCO Documents used" or "CCO LINKS". Prior to closing a case, the TAC engineer will have to put in links to the documents he used in his resolution. This will enable later TAC engineers reviewing the case at a later time to come to a resolution in a very short time since they will not have to search for the right documents for as long. They know beforehand that the Cisco documents have the answers they hope for. Today, based on the engineer’s behavior towards documenting the cases, tickets may or may not include links used to solve an issue. The suggestion is to make this common practice and educate all TAC engineers on the importance of filling out the field of links used before a case is closed. These links can be configuration guides, features, release notes, field notices or any Cisco written documents. Cost: Many of the suggested changes will not cost any money. Every team manager will need to send an email to his team explaining our new behavior toward customer tickets and what kinds of updates are necessary prior to closing a case. Some of the changes are going to be low cost. Those are the ones we suggested to make changes to the ticket structure and add certain fields to the ticket. The ticketing system is definitely open to additions and we field changes all the time, hence the cost to add extra fields will more than likely be minimal. Measuring results: This is a big project and a long term project with the goal of enriching a database and an information center "topic database" that has been around for years. With the right implementation and commitment from TAC engineers we can see results in about 2-3 months. Based on the number of tickets we take in everyday, I believe that 3 months is enough time to have accumulated a great number of tickets covering all of the issues on a given technology, and at the end of the 3 months period we will have at least two or three well documented tickets with all the troubleshooting commands and useful links allowing engineers to gather a lot of information from just reviewing those rich cases. Read More