Sitemap

kc@example.com (kc kim) — Sat, 17 Apr 2021 08:32:00 +0900

What Is a Sitemap?

A sitemap is a file that helps search engines such as Google and Naver comprehensively index a site. Basically, it can be thought of as a file that lists URLs. Bots crawl the site based on this file.

Content types and update frequency can be specified, but the most important thing is where sitemap.xml is located. Since only URLs under the domain below sitemap.xml are crawled, the installation location must be chosen carefully. In general, it is best to place it at the root.

Sitemap XML Format

<?xml version="1.0" encoding="UTF-8"?>
<urlset>
  <url>
    <loc>https://www.devkuma.com/docs/java/static/</loc>
    <lastmod>2022-04-03T20:41:00+09:00</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.5</priority>
   </url>
</urlset>

Tag	Required/Optional	Description
`<urlset>`	Required	The tag that wraps the whole document and refers to the current protocol standard.
`<url>`	Required
`<loc>`	Required	Page URL. It must include a trailing slash and the value must be 2,048 characters or fewer.
`<lastmod>`	Optional	Last update date of the file.
`<changefreq>`	Optional	Page update frequency.
`<priority>`	Optional	URL priority. A value from 0.0 to 1.0 can be specified. The default is 0.5. Do not set high priority for every URL on the site.

Page update frequency (changefreq) list:

always: contents are updated every time the page is accessed
hourly: once per hour or less
daily: at least once per day
weekly: at least once per week
monthly: at least once per month
yearly: at least once per year
never: crawled periodically, for pages that do not need updates

When Using Multiple Sitemap Files

If there are 50,000 or more URLs, multiple sitemaps are needed. In that case, create a sitemap index file and tell crawlers that multiple sitemaps exist.

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
   <sitemap>
      <loc>https://www.devkuma.com/sitemap1.xml.gz</loc>
      <lastmod>2022-12-06T01:57:17+09:00</lastmod>
   </sitemap>
   <sitemap>
      <loc>https://www.devkuma.com/sitemap2.xml.gz</loc>
      <lastmod>2021-01-01</lastmod>
   </sitemap>
</sitemapindex>

Tag	Required/Optional	Description
`<loc>`	Required	Sitemap file name
`<lastmod>`	Optional	Last update date of the file

References

sitemaps.org

robots.txt

kc@example.com (kc kim) — Sat, 17 Apr 2021 08:32:00 +0900

How Search Engines Originally Work

It is the Robots Exclusion Protocol.
A robot called a crawler travels around the Internet and collects site information.
The indexer analyzes the information collected by the crawler.
Based on the analyzed data, each search engine returns search results according to its algorithm.

What Is robots.txt?

robots.txt is a text file that tells crawlers which pages to crawl or not crawl.
It is published in the top-level directory of the domain.
robots.txt is still a recommendation, so there is no absolute obligation to follow it.

robots.txt Format

User-agent: search bot name
Allow: access permission setting, available only for Googlebot
Disallow: access blocking setting
Crawl-delay: delay before the next visit, in seconds
Sitemap: sitemap specification

robots.txt Examples

Allow all search bots to access all documents

User-agent: *
Allow: /

* means all robots, and / means all directories.

Block all search bots from all documents

User-agent: *
Disallow: /

Allow access to a specific directory

User-agent: Googlebot
Allow: /foo/bar/

Block access to a specific directory

User-agent: Googlebot
Disallow: /foo/bar/

Allow only Googlebot and block all others

User-agent: Googlebot
Allow: /

User-agent: *
Disallow: /

Expose Only Part of a Homepage Directory to Search Engines

 User-agent: * 
 Disallow: /conection/ 
 Disallow: /my_conection/

Block Part of a Homepage Directory from Search Engines

User-agent: *
Disallow: /my_page/

Site Load and Performance Perspective

If crawler visits increase site load, unimportant large amounts of content can be removed from crawler traversal using robots.txt, reducing site load and improving crawl efficiency for important content.

Separating important content from unimportant content is also beneficial for SEO and site load.

Unimportant content may include:

Pages that do not need to be indexed by search engines
Low-value content pages
Multiple pages with identical content
Landing pages for ads placed on the site
Pages you want to make available only to limited people
Management system files

Security Perspective

If “pages to crawl” or “pages not to crawl” are set in robots.txt, content intended only for limited people may become visible.

If management system files or pages intended for limited disclosure are set in robots.txt, they may not appear in search engine results, but they are still publicly exposed through robots.txt. In other words, files related to management or pages meant only for limited people can be revealed.

Using robots.txt can reduce the risk of appearing in search results, but if security-sensitive content is exposed in robots.txt, a security risk occurs.

Therefore, files related to security-sensitive management or pages meant only for specific limited people must have reliable access restrictions such as login authentication or IP address restrictions.

devkuma – SEO

Sitemap

What Is a Sitemap?

Sitemap XML Format

When Using Multiple Sitemap Files

References

robots.txt

How Search Engines Originally Work

What Is robots.txt?

robots.txt Format

robots.txt Examples

Expose Only Part of a Homepage Directory to Search Engines

Block Part of a Homepage Directory from Search Engines

Site Load and Performance Perspective

Security Perspective

References