Specialized Search engine marketing: Technical Search engine optimisation refers to Web-site and server optimization that assists the crawler in crawling, indexing and ranking functions to rank your site greater. txt file is then parsed and can instruct the robotic concerning which internet pages are certainly not to generally be crawled. https://www.youtube.com/watch?v=0xsCkQ6bbjk