Personalized Search Engines - Research Paper Example

Profile-Based Web-Searching (Computer Science Project Report) Research Methods & Professional Issues Table of Contents Introduction 1.1 Project Background and Aim 1.2 Project objectives 1.3 Project Deliverables 1.4 Project Plan 2. Literature Review 2.1 Preliminary Analysis of Some Personalized Search Engines 2.1.1 Google Personalized Search 2.1.2 Yahoo Personalized Search 2.1.3 MSN (or Windows Live Search) Personalized Search 2.2 Personalized Search Engines comparison 2.3 Methodology 2.3.1 The Prototyping Model 2.3.2 Database model design 2.4 Software/Tools and Modelling Techniques Conclusion References Appendix Chapter 1 Introduction Search engines are more and more important in our life, as the search technology develops, the personalized search has become expected by search Internet users. It is the fine-tuning of search results and advertising, based on an individual’s information, demographic, performance and other factors that has evolved into the search engines of today. A search engine understands Internet users better, because a search engine can target search results, preferences, advertising and sponsor links. 1.1 Project Background and Aim Providing profile-based Web searching to users was an emerging concept, first introduced by Eurekster in 2004. At that time it was biggest surprise that Google, Yahoo and Microsoft (MSN now Windows Live Search) had not introduced this concept initially, allowing a smaller company, when compared to Google, Yahoo and Microsoft, had introduced this concept of a personalized search. In case of a personalized search various types of information from users is collected and based on that information the search content is filtered and displayed to users. The Internet (Web) contains lots of information and by searching, even using specific keywords, it is a possibility that irrelevant information will be returned. Now search engines are working to improve searches, allowing the solution of personalized searches, to give its users search results based on the users demographic information, interests and preferences. In addition a users search history can also be maintained by search engines, for easier repeated searches. The study of “Profile-Based Web-Searching” will provide an opportunity to learn how different personalized search engines work and make use of user specific information for assisting web searching process. 1.2 Project objectives The research objectives of the projects are: Produce a preliminary web searching engine system based on user profiling. Test and evaluate the web searching engine system in terms of requirements and deliverables. To deepen the knowledge of software engineering. To gain a better understanding of PHP combined with MySQL and Navicat technology. To have a better understanding of modeling technology. Fully describe the software development process in a scientific report. Gain the experience of software development and management. 1.3 Project Deliverables Basic Deliverables Study existing methods for user-profiling methods for web searching and layout a design for implementation. Study existing technologies for interface and database design. Implement a preliminary web searching system based on the design. Intermediate Deliverable Refine the set of metrics and incorporate the user-profiling method in the web searching system. Evaluate the effectiveness of the method. Advanced Deliverable Refine the user-profiling method, the searching mechanism and the system to further improve its effectiveness through comparison with other methods. 1.4 Project Plan Various phases for Profile Based Web Searching Weeks March July August 1 2 3 4 5 6 7 8 9 10 11 12 Problem Identification Information gathering and Analysis Logical and conceptual design Physical design Physical implementation Testing , Review and Changes Implementation Maintenance Figure 1: Project Plan - Chapter 2 Literature Review According to Johnson (2005), personalized search drives financial benefits both in short term and long term. In the short term, advertisers increase revenue by targeting ads and promotions at the users who are most likely to click and buy. In addition, consumers want personalization and are more willing to stay with sites and services that ‘know’ them and respond to their specific lifestyles and preferences. Moreover, sites that employ personalized search automatically filter the Web for consumers based on their individual preferences and saves consumers time and frustration, and bonds users to the site. The techniques used in personalizing Web content are used to Personalized Web Search. According to Johnson (2005), there are many ways, but they all require ability that is discussed here. The first thing that a search engine requires is the preferences and demographic information of the users. This can be achieved by search engines in such a way that user’s Online experience is not interfered. The next thing is the determination of the type of content and advertising that are the best predictors of relevance for the customers. Now the content that is to be displayed to users are filtered based on the previous information. In the last step these results are presented to user in a way that is easy for the consumers to understand. According to Johnson (2005), the end result of applying personalization to search technology is an enhanced search experience with following considerations. The first thing is noticed that the sponsored links that are displayed to users should be based on their tastes, interests, and recent search histories. This can increase the likelihood of click through to advertisements and other content. Also one more important thing is taken care here is of advertising ads based on user preferences as well as terms entered by users. The last thing that can be done here is to show Web search results that can be organized by style, meaning, category, etc. These results can be presented under a “My Results” tab so as not to displace unfiltered results. 2.1 Preliminary Analysis of Some Personalized Search Engines Eurekster had launched the concept of personalized search engine in 2004. Neither Google nor Yahoo introduced this service, but instead from Eurekster this service launched, which was opens to the general public on 21 January 2004. Before launching this service the Eurekster site had been involved in a beta test involving only a few hundred people for couple of months. Eurekster provides many options and some of them are searches filtered by friends, SearchMates, sharing searches, other sites, and using any search engines in combination with Eurekster (Sullivan, 2004). In recent years, three big search providers Google, Yahoo and Microsoft have taken over to dominate personalized searches. Each of the three major search engines use an algorithm. Google uses the Google Hilltop algorithm. Yahoo uses a page rank algorithm, even though it does not always satisfy customers. MSN (Windows Live Search) uses the a keyword based search algorithm. These algorithms will be explained in detail below. Jeh and Widom (2003, 272) explain: The fundamental motivation underlying PageRank is the recursive notion that important pages are those linked-to by many important pages. A page with only two in-links, for example, may seem unlikely to be an important page, but it may be important if the two referencing pages are Yahoo! and Netscape, which themselves are important pages because they have numerous in-links. One way to formalize this recursive notion is to use the ``random surfer model introduced in [11]. Imagine that trillions of random surfers are browsing the web: if at a certain time step a surfer is looking at page P, at the next time step he looks at a random out-neighbor of P. As time goes on, the expected percentage of surfers at each page P converges (under certain conditions) to a limit that is independent of the distribution of starting points. Intuitively, this limit is the PageRank of P, and is taken to be an importance score for P, since it reflects the number of people expected to be looking at P at any one time. If P equals page, then r equals rank. PageRank is p(r). These are the basis of most algorithms using page rank. 2.1.1 Google Personalized Search On March 29, 2004, Google provided a new tool that would help people make their personalized web searching. This Google Personalized Search has allowed people to create a profile of their interests, which then influences the Web site links shown when they conduct a search and was available for testing at the Google Labs. (Bazeley, 2004) Google personalized search left the Google lab and made available to users on 38 domains in addition to google.com on November 2005. Googles personalized searches give more weight to topics that interest one and provided a feature to maintain history of searches on Google, allowing revisiting pages previously visited just by scanning the history of the search. People who use this service, have to sign up for a Google account such as Gmail, AdSense, and other Google services, like AOL (Sherman, 2005). On February 2, 2007, Google enhanced its personalized search. Now anyone who signs-up for any Google service using a Google Account (such as Gmail, AdSense, Google Analytics among others) will automatically enroll into three additional Google products: Search History, Personalized Search, and Personalized Homepage (Sullivan, 2007). Screenshot 1, 2 and 3 show the personalized search history of Google Personalized Search (Appendix). The Google Hilltop algorithm powers personalized searches on Google. The Google Hilltop algorithm is defined (Thibodeau, 2003) as: the Google Hilltop algorithm determines the relevance and importance of a specific web page determined by the search query or keyword used in the search box. In its basic, simplest form, instead of relying only on the PageRank™ value to find “authoritative pages”, it would be more useful if that “PR value” (PageRank™ value) would be more relevant by the topic or subject of that same page. PageRank and key words are used in Google personalized searches. This algorithm is the most popular. 2.1.2 Yahoo Personalized Search In May 2004, Yahoo provided a personal search product that allows users to save bookmarks in a personal Online space and append notes to them. Users can save and share each search results with others via e-mail, RSS feeds or Myyahoo "modules" that can be sent to other Yahoo users (Bazeley 2004). Presently Yahoo offers MyWeb 2.0 (Beta) Personalized Search, which is shown in Screenshot 4 and 5 (Appendix). Yahoo uses the page rank algorithm, even on personalized searches. The following chart shows Yahoos search for the word “spoon”. The first query was the word "spoon", without quotes. The picture on the left clearly displays what it turned up, but as you probably arent psychic, its a good idea to explain the meanings of those colors, dots and lines. Dots: Light green dots () display the PageRank of individual sites. For example, the site displayed first in the results had a PageRank of 7. Thus, the light green dot on the very left side of the table is placed on the fourth line from the top. From the PageRank scale on the left you can clearly see that the dot is in the right place. Light blue dots () display whether the site had the keyword in its title or not. If the keyword was in the title, the dot is placed near the top of the table. If no keyword was found in the title, the dot is placed near the bottom. As you can see, all the sites in the top ten had the keyword in their titles and thus all the light blue dots are located at the top of the table. Light yellow dots () display whether the site had the keyword in its description. The sites that had the keyword in the description are marked by an yellow dot placed at the top of the table, while those that didnt can be distinguished from the dot being located at the bottom of the table. By quickly looking at the table, you can see that from the sites in the top 10 for this keyword, only one had the keyword in the description while nine sites didnt. Light grey dots () display whether the site had the keyword in its URL. Should the URL contain a keyword, the dot is placed at the top of the table. Should it not, the dot can once again be found near the bottom. Now, take a good look at the top part of the table, right were the dots representing the second site are located. See how the dot for the URL is actually above the table? The site in question happened to have two occurrences of the keyword in its URL, but as you can see, the scale on the right only goes up to 1.00. If the amount of keywords in the URL, description or title exceeds one, Ive placed the dot "out of scale". This should help you notice the difference between the sites that have a description/URL/title that only contains the keyword once and those that have multiple keywords in those places. Light red dots () display whether the site had the keyword in the name of the category it is listed in. This dot follows the same rules as the three previous dots, so there probably is no need to discuss it further. (A Promotional Guide, 2007) 2.1.3 MSN (or Windows Live Search) Personalized Search Microsoft had recently launched Windows Live Search (previously MSN Search). In which various features that can be personalized according to need of the users can be added. Screenshots of this are shown in Screenshot 6, 7 and 8 (Appendix). MSN (Windows Live Search) uses a keyword based search algorithm. Perhaps the first and probably most important aspect of the MSN Search Beta is the need for good keyword rich content. Relevance of theme and topic appears to be very important to MSN, as it is becoming for Google optimization. There are also additional considerations for levels of on page keyword density that differ somewhat from Google. On the other hand, MSN seems to strongly dislike keyword stuffing, and will drop a site’s ranking accordingly. (Hurlbert, 2004) Even if a website has what a searcher needs, MSN (Windows Live Search) will not rank it high if keyword stuffing is present. This service is still not giving customers what they need, like Google does. 2.2 Personalized Search Engines comparison Figure 2 shows important features of all the three most popular search engines. These features had been added to these search engines after various research from a large development team, who always tries to find new features that users want to see and that can help the users to conduct their searching more easily, finding more relevant information from the web faster. In project “Profile-Based Web-Searching”, initially different common features that are used by these search engines will be incorporated. After that some of the common features that are used for personalized search, such as history and bookmark, will be included in “Profile-Based Web-Searching” that will be based on user specific information. The main limitation of the “Profile-Based Web-Searching” is that it is currently based on the existing search engines. In addition, it is college level project and developed by a single developer. Features Google Search (http://www.google.com) Windows Live Search (MSN) (http://www.live.com/) Yahoo Search (http://search.yahoo.com/) File Types Web (html), pdf, msoffice, mac, corel, swf, rss, xml Web (html), ppt, xls, doc, pdf Web, ppt, xls, doc, pdf, xml, rss, rdf, swf Search Levels Simple and Advanced Simple and Advanced Simple and Advanced Boolean Operators +, -, OR, (AND by Default) AND, OR, NOT, +, - +, -, (AND by Default) Truncation/ stemming Automatically stems some words and looks for word variants. No No Restrict by date Advanced Search Advanced: Results Ranking -move slider Advanced - time periods Field Search Advanced: Filters for Title, Format, URL, Domain and Site Advanced: Specify domain, country, and language. Advanced Form: words in title or url. Domain file format Proximity Can use * as a wildcard in a phrase Search Builder: Results Ranking – move slider to “exact match” Can use * as a wildcard in a phrase (Not as good as at Google) Sorting Relevance, site Relevance, site, sliders Relevance, site Cached Page Yes (with date) Yes (with date) Yes Search Aids Spells check Definitions of search terms from Answers.com Recognizes US addresses - map search. US Phonebook search Other number searches SafeSearch Filtering Language Display Translation Spelling Correction Encarta Spell check Shortcuts - dictionary, encyclopedia, travel, weather. Instant search answers while you type. Personal Search Tools Search History and Personalized Search and also Google Notebook for saving clips. Windows Live Favorites - online bookmarks Search Macros – build your own or use pre-built MyWeb2 - save and share pages and bookmarks. Verticals & Tools Search Alerts, Image Search , Video Search (TV, educational, personal), News Search and News Alerts, Newsgroup Search ,Local search , Google Maps for locating places. Scholar - journals and scholarly sites, Books - snippets and some full text, News Archive, More through Google Labs and Other Google Services Images, Video Feeds: Search for and subscribe to. Local and Maps - birds eye view on maps for US and UK Live Academic – scholarly Preferences - lists settings for all search services, Images, Video, Audio, Local and Maps, News, Answers – questions, Subscriptions - access for for-fee databases. All Yahoo Services Yahoo Tools for Researchers Toolbar Google Toolbar and Desktop Search Windows Live Toolbar Yahoo Companion Subject Directory Open Directory Project Ranked None Yahoo Directory Sponsored Links Google AdWords Yahoo Marketing and MSN Ad Center Yahoo Marketing Figure 2: Search Engine Comparison Chart Source: http://www.websearchguide.ca/research/compfram.htm 2.3 Methodology Primary as well as secondary research will be used for “Profile-Based Web-Searching”. There are various methods available based on any software that can be developed. Some models, which are the Waterfall Model, the Agile Model, the RAD Model and the Prototype Model, can be used. For this project, the Prototype model (Show figure 3) will be used and it is described below: 2.3.1 The Prototyping Model It is a working model, which is functionally like a component of the product. It supports a general view of what is expected from the software product to client. By allowing the client to interact and experiment with a working illustration of the product, it shows the flexibility of the development process that is attempted to increase. If the client is happy with the functioning of the prototype, the developmental process only continues once. Meanwhile, the specifications of the client’s actual requires are determined by the developer at this stage. Advantages of Prototyping: Developing fast and easier for end users to learn/use. Also low cost for development. Development backlog decreases and less change needed after implementation. Users know what to anticipate at implementation as end-user involvement. End-user/analyst communication requirements easier to determine. Disadvantages of prototyping: The overall quality of the software was not considered in the rush to develop the prototype, which was failed to realize by clients. Compromise the quality of the product and the real purpose of the prototype may not be a focus by developers. When the developer has fulfilled his/her duties, but the client does not agree, the Prototyping Model will hardly be acceptable. Figure 3: Prototyping Cycle Source (modified): www.cs.pitt.edu/~chang/153/c02process/s32.gif 2.3.2 Database model design According to Powel (2006), there are various methodologies available for designing database models. Each of these different approaches consists of a number of steps. These separate steps are interchangeable, repeatable, and iterative. The following sequence of steps to database model design seems the most sensible and will be used with the prototype model. Requirements analysis: It is the first phase in which information about the nature of the data, features required, and any specialized needs will be covered. Conceptual design: This part will be done with the help of graphical tools (for example Rational Rose.) for making Entity Relationship Diagrams (ERDs). This will be made based on data flow diagram and use case diagram that is made in analysis part of the project. Logical design: Create database language commands to generate table definitions. Physical design: Adjust database language commands to alter the database model for the underlying physical attributes of tables. Tuning Phase: Tuning phase includes appropriate indexing, further normalization, or even de-normalization and security features (Powel 2006, p.20). 2.4 Software/Tools and Modeling Techniques Technologies Used: Various technologies, tools, and languages that will be used for this project are summarized below: Modelling Tool/Language Unified Modeling Language (UML) Front End Technology: PHP Development Tool/Editor: Macromedia Dreamweaver Server Side Scripting Language: PHP Client Side Scripting Language: JavaScript Database Management System RDBMS: MySQL Front End for MYSQL: Navicat 2005 or Latest Server Internet Server: PHP Environment: Windows NT UML The Unified Modeling Language (UML) is the standard modeling language for software and systems development. It is made up of nine diagrams that can be used to model a system at different points of time in the software life cycle of a system. These diagrams fall in various categories, mainly static, dynamic and implementation (Chitnis & Tiwari etc., 2006). Perl Hypertext Preprocessor (PHP) PHP is a kind of software which was released by Free Software foundation freely, which was designed as a tool for producing dynamic web pages in the form of programming language, for example severing side scripting languages along with JavaScript as client side scripting language. Meanwhile, the functions of PHP can be defined as three key features as server-side scripting, command-line scripting and client side GUI applications. In addition, there are number of free and open sources libraries within the core build of PHP. This software functions similar with the Perl language. The current PHP server version is PHP 5.2, which was released on November 2, 2006. It is mostly used with MySQL RDBMS. MySQL MySQL is the database which supports the entire Google search engine, which is widely known as the most popular open source database in the world. In order to ensure all the transactions follow the ACID model, almost all the commercial RDBMS functionality are supported by MySQL which allows supporting standard data types, building of indexes and database replication among other features. The Perl Hypertext Preprocessor (PHP), which is dynamic web development language, is completely cooperating with MySQL. Dreamweaver Dreamweaver is a web development tool that enables users to efficiently design, develop and maintain standards-based website and applications. It supports various languages and technologies such as ASP, ASP.NET, PHP, JSP, CSS, HTML, XHTML and XML. It provides option at the time of development in design view, code view and both by splitting window. Navicat (Database administration MySQL Client) In practice, the important functions of a database administrator are control, recovery, and performance measurement, which can be completed by Navicat through a single interface which is a client interface for MySQL database. This powerful software can eliminate time-consuming data entry and the errors that accompany it, and moreover, convert DBF, CSV, text, or XML files to MySQL databases. Back-up/restore databases, data transfer, import/export wizard, visual query builder, batch job scheduling, data synchronization, and SSH tunnel are seen as major functions of MySQL admin client. The access to multiple connections to local and remote databases under the style of Windows Explorer, cascading open to databases, tables and ultimately data are the primary applications window allows. Conclusion Personalized search drives financial benefits both in short term and long term. These days many companies such as Yahoo, Google and Microsoft are involved in giving its user personalized search option, in which user can check history, bookmarks for various searches made. This project is also intended to provide to make use of user specific information to assist the web searching process. In this paper, various existing personalized search engines are covered and various technologies and tools that can be used for development of such a system are discussed. References A Promotional Guide. (2007) The Ranking Algorithm of Yahoo. http://www.apromotionguide.com/yahooalgo.html, (Accessed on 10 May 2007). Adobe. (2007) http://www.adobe.com/products/dreamweaver/, (Accessed on 21 February 2007). Bazeley, M. (2004) Google to Enable Personalized Searches. San Jose Mercury News (CA); 03/30/2004. Bazeley, M. (2004) Yahoo delivers personal search technology. San Jose Mercury News (CA); 10/05/2004. Chitnis, M., Tiwari, P. & Ananthamurthy, L. (2006) UML Overview. http://www.developer.com/design/article.php/1553851, (Accessed on 25 February 2007). Hurlbert, W. (2004) MSN Optimization. http://www.seochat.com/c/a/MSN- Optimization-Help/MSN-Search-Engine-Beta-Optimization-Techniques/, (Accessed on 10 May 2007). Jeh, G. and J. Widom. (2003) Scaling Personalized Web Search. http://infolab.stanford.edu/~glenj/spws.pdf, (Accessed on 25 May 2007). Johnson, S. (2005) Personalized Search, January 27, 2005. http://www.imediaconnection.com/content/4977.asp, (Accessed on 21 February 2007). Navicat (Database administration – MySQL Client) 6.3.5. (2007) http://www.apple.com/downloads/macosx/development_tools/ navicatdatabaseadministrationmysqulclient.html, (Accessed on 21 February 2007). Powel, G. (2006) Beginning Database Design. Wiley Publishing, Inc., United States of America. p20. Sherman, C. (2005) Google Personalized Search Leaves Google Labs, November 10, 2005. http://searchenginewatch.com/showPage.html?page=3563036, (Accessed on 23 February 2007). Sullivan, D. (2004) Eurekster Launches Personalized Social Search, January 21, 2004. http://searchenginewatch.com/showPage.html?page=3301481, (Accessed on 21 February 2007). Sullivan, D. (2007) Google Ramps Up Personalized Search, February 2, 2007. http://searchengineland.com/070202-224617.php, (Accessed on 23 February 2007). Thibodeau, S. (2003) The Google Hilltop Algorithm. http://www.rankforsales.com/search-engine-algorithms/google-hilltop- algorithm.html, (Accessed on 10 May 2007). Web Search Guide (2007) Search Engine Comparison Chart: Google, Windows Live Search, Yahoo (April 2, 2007). http://www.websearchguide.ca/research/compfram.htm, (Accessed on 10 April 2007). Unknown (n.d.) http://www.cs.pitt.edu/~chang/153/c02process/s32.gif, (Accessed on 25 February 2007). Appendix Some Screenshots of Existing Personalized Search Engines Screenshot 1: Creating an Google Account Screenshot 2: Sign-in to Google Account Screenshot 3: Google Search History Page Screenshot 4: Sign-in to Yahoo Search (My Web) Account Screenshot 5: Yahoo Search (My Web) Page Screenshot 6: Windows Live (MSN) Search Page Sign-in Screenshot 7: Windows Live (MSN) Search Page1 Screenshot 8: Windows Live (MSN) Search Page 2 Read More

2.1 Preliminary Analysis of Some Personalized Search Engines Eurekster had launched the concept of personalized search engine in 2004. Neither Google nor Yahoo introduced this service, but instead from Eurekster this service launched, which was opens to the general public on 21 January 2004. Before launching this service the Eurekster site had been involved in a beta test involving only a few hundred people for couple of months. Eurekster provides many options and some of them are searches filtered by friends, SearchMates, sharing searches, other sites, and using any search engines in combination with Eurekster (Sullivan, 2004).

In recent years, three big search providers Google, Yahoo and Microsoft have taken over to dominate personalized searches. Each of the three major search engines use an algorithm. Google uses the Google Hilltop algorithm. Yahoo uses a page rank algorithm, even though it does not always satisfy customers. MSN (Windows Live Search) uses the a keyword based search algorithm. These algorithms will be explained in detail below. Jeh and Widom (2003, 272) explain: The fundamental motivation underlying PageRank is the recursive notion that important pages are those linked-to by many important pages.

A page with only two in-links, for example, may seem unlikely to be an important page, but it may be important if the two referencing pages are Yahoo! and Netscape, which themselves are important pages because they have numerous in-links. One way to formalize this recursive notion is to use the ``random surfer model introduced in [11]. Imagine that trillions of random surfers are browsing the web: if at a certain time step a surfer is looking at page P, at the next time step he looks at a random out-neighbor of P.

As time goes on, the expected percentage of surfers at each page P converges (under certain conditions) to a limit that is independent of the distribution of starting points. Intuitively, this limit is the PageRank of P, and is taken to be an importance score for P, since it reflects the number of people expected to be looking at P at any one time. If P equals page, then r equals rank. PageRank is p(r). These are the basis of most algorithms using page rank. 2.1.1 Google Personalized Search On March 29, 2004, Google provided a new tool that would help people make their personalized web searching.

This Google Personalized Search has allowed people to create a profile of their interests, which then influences the Web site links shown when they conduct a search and was available for testing at the Google Labs. (Bazeley, 2004) Google personalized search left the Google lab and made available to users on 38 domains in addition to google.com on November 2005. Googles personalized searches give more weight to topics that interest one and provided a feature to maintain history of searches on Google, allowing revisiting pages previously visited just by scanning the history of the search.

People who use this service, have to sign up for a Google account such as Gmail, AdSense, and other Google services, like AOL (Sherman, 2005). On February 2, 2007, Google enhanced its personalized search. Now anyone who signs-up for any Google service using a Google Account (such as Gmail, AdSense, Google Analytics among others) will automatically enroll into three additional Google products: Search History, Personalized Search, and Personalized Homepage (Sullivan, 2007). Screenshot 1, 2 and 3 show the personalized search history of Google Personalized Search (Appendix).

The Google Hilltop algorithm powers personalized searches on Google. The Google Hilltop algorithm is defined (Thibodeau, 2003) as: the Google Hilltop algorithm determines the relevance and importance of a specific web page determined by the search query or keyword used in the search box. In its basic, simplest form, instead of relying only on the PageRank™ value to find “authoritative pages”, it would be more useful if that “PR value” (PageRank™ value) would be more relevant by the topic or subject of that same page.

Read More