Malik Murtaza Mushtaq

27/07/2023

✌

Check out Malik Murtaza Mushtaq's video.

09/05/2022

Indexation-controlling parameters

• Noindex: Tells a search engine not to index a page.

• Index: Tells a search engine to index a page. Note that you don't need to add this meta tag; it's the default.

• Follow: Even if the page isn't indexed, the crawler should follow all the links on a page and pass equity to the linked pages.

• Nofollow: Tells a crawler not to follow any links on a page or pass along any link equity.

12/04/2022

Types of Robots Meta Directives

There are two types of robots meta directives: those that are part of the HTML page (like the meta robots tag) and those that the web server sends as HTTP headers (such as x-robots-tag). The same parameters (i.e., the crawling or indexing instructions a meta tag provides, such as "noindex" and "nofollow" in the example above) can be used with both meta robots and the x-robots-tag; what differs is how those parameters are communicated to crawlers.

11/04/2022

Meta Directives

Meta directives give crawlers instructions about how to crawl and index information they find on a specific webpage. If these directives are discovered by bots, their parameters serve as strong suggestions for crawler indexation behavior. But as with robots.txt files, crawlers don't have to follow your meta directives, so it's a safe bet that some malicious web robots will ignore your directives.

10/04/2022

Robots Meta Tags

Robots meta directives (sometimes called "meta tags") are pieces of code that provide crawlers instructions for how to crawl or index web page content. Whereas robots.txt file directives give bots suggestions for how to crawl a website's pages, robots meta directives provide more firm instructions on how to crawl and index a page's content.

09/04/2022

SEO best practices

Do not use robots.txt to prevent sensitive data (like private user information) from appearing in SERP results. Because other pages may link directly to the page containing private information (thus bypassing the robots.txt directives on your root domain or homepage), it may still get indexed. If you want to block your page from search results, use a different method like password protection or the noindex meta directive. Some search engines have multiple user-agents. For instance, Google uses Googlebot for organic search and Googlebot-Image for image search. Most user agents from the same search engine follow the same rules so there's no need to specify directives for each of a search engine's multiple crawlers, but having the ability to do so does allow you to fine-tune how your site content is crawled

07/04/2022

SEO best practices

• Make sure you're not blocking any content or sections of your website you want crawled.

Links on pages blocked by robots.txt will not be followed. This means 1.) Unless they're also linked from other search engine accessible pages (i.e. pages not blocked via robots.txt, meta robots, or otherwise), the linked resources will not be crawled and may not be indexed. 2.) No link equity can be passed from the blocked page to the link destination. If you have pages to which you want equity to be passed, use a different blocking mechanism other than robots.txt.

06/04/2022

How to create a robots.txt file?
TXT
If you found you didn't have a robots.txt file or want to alter yours, creating one is a simple process. This article from Google walks through the robots.txt file creation process, and this tool allows you to test whether your file is set up correctly.

05/04/2022

Why do you need robots.txt?
Robots.txt files control crawler access to certain areas of your site. While this can be very dangerous if you accidentally disallow Googlebot from crawling your entire site (!!), there are some situations in which a robots.txt file can be very handy. Some common use cases include:
Preventing duplicate content from appearing in SERPs (note that meta robots is often a better choice for this)
Keeping entire sections of a website private (for instance, your engineering team's staging site)
Keeping internal search results pages from showing upon a public SERP
Specifying the location of sitemap(s) Preventing search engines from indexing certain files on
your website (images, PDFs, etc.)
Specifying a crawl delay in order to prevent your servers