devkuma – Web

Sitemap

kc@example.com (kc kim) — Sat, 17 Apr 2021 08:32:00 +0900

What Is a Sitemap?

A sitemap is a file that helps search engines such as Google and Naver comprehensively index a site. Basically, it can be thought of as a file that lists URLs. Bots crawl the site based on this file.

Content types and update frequency can be specified, but the most important thing is where sitemap.xml is located. Since only URLs under the domain below sitemap.xml are crawled, the installation location must be chosen carefully. In general, it is best to place it at the root.

Sitemap XML Format

<?xml version="1.0" encoding="UTF-8"?>
<urlset>
  <url>
    <loc>https://www.devkuma.com/docs/java/static/</loc>
    <lastmod>2022-04-03T20:41:00+09:00</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.5</priority>
   </url>
</urlset>

Tag	Required/Optional	Description
`<urlset>`	Required	The tag that wraps the whole document and refers to the current protocol standard.
`<url>`	Required
`<loc>`	Required	Page URL. It must include a trailing slash and the value must be 2,048 characters or fewer.
`<lastmod>`	Optional	Last update date of the file.
`<changefreq>`	Optional	Page update frequency.
`<priority>`	Optional	URL priority. A value from 0.0 to 1.0 can be specified. The default is 0.5. Do not set high priority for every URL on the site.

Page update frequency (changefreq) list:

always: contents are updated every time the page is accessed
hourly: once per hour or less
daily: at least once per day
weekly: at least once per week
monthly: at least once per month
yearly: at least once per year
never: crawled periodically, for pages that do not need updates

When Using Multiple Sitemap Files

If there are 50,000 or more URLs, multiple sitemaps are needed. In that case, create a sitemap index file and tell crawlers that multiple sitemaps exist.

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
   <sitemap>
      <loc>https://www.devkuma.com/sitemap1.xml.gz</loc>
      <lastmod>2022-12-06T01:57:17+09:00</lastmod>
   </sitemap>
   <sitemap>
      <loc>https://www.devkuma.com/sitemap2.xml.gz</loc>
      <lastmod>2021-01-01</lastmod>
   </sitemap>
</sitemapindex>

Tag	Required/Optional	Description
`<loc>`	Required	Sitemap file name
`<lastmod>`	Optional	Last update date of the file

References

sitemaps.org

robots.txt

kc@example.com (kc kim) — Sat, 17 Apr 2021 08:32:00 +0900

How Search Engines Originally Work

It is the Robots Exclusion Protocol.
A robot called a crawler travels around the Internet and collects site information.
The indexer analyzes the information collected by the crawler.
Based on the analyzed data, each search engine returns search results according to its algorithm.

What Is robots.txt?

robots.txt is a text file that tells crawlers which pages to crawl or not crawl.
It is published in the top-level directory of the domain.
robots.txt is still a recommendation, so there is no absolute obligation to follow it.

robots.txt Format

User-agent: search bot name
Allow: access permission setting, available only for Googlebot
Disallow: access blocking setting
Crawl-delay: delay before the next visit, in seconds
Sitemap: sitemap specification

robots.txt Examples

Allow all search bots to access all documents

User-agent: *
Allow: /

* means all robots, and / means all directories.

Block all search bots from all documents

User-agent: *
Disallow: /

Allow access to a specific directory

User-agent: Googlebot
Allow: /foo/bar/

Block access to a specific directory

User-agent: Googlebot
Disallow: /foo/bar/

Allow only Googlebot and block all others

User-agent: Googlebot
Allow: /

User-agent: *
Disallow: /

Expose Only Part of a Homepage Directory to Search Engines

 User-agent: * 
 Disallow: /conection/ 
 Disallow: /my_conection/

Block Part of a Homepage Directory from Search Engines

User-agent: *
Disallow: /my_page/

Site Load and Performance Perspective

If crawler visits increase site load, unimportant large amounts of content can be removed from crawler traversal using robots.txt, reducing site load and improving crawl efficiency for important content.

Separating important content from unimportant content is also beneficial for SEO and site load.

Unimportant content may include:

Pages that do not need to be indexed by search engines
Low-value content pages
Multiple pages with identical content
Landing pages for ads placed on the site
Pages you want to make available only to limited people
Management system files

Security Perspective

If “pages to crawl” or “pages not to crawl” are set in robots.txt, content intended only for limited people may become visible.

If management system files or pages intended for limited disclosure are set in robots.txt, they may not appear in search engine results, but they are still publicly exposed through robots.txt. In other words, files related to management or pages meant only for limited people can be revealed.

Using robots.txt can reduce the risk of appearing in search results, but if security-sensitive content is exposed in robots.txt, a security risk occurs.

Therefore, files related to security-sensitive management or pages meant only for specific limited people must have reliable access restrictions such as login authentication or IP address restrictions.

References

Web

kc@example.com (kc kim) — Sun, 03 Apr 2022 18:02:00 +0900

What Is the Web?

The term most often encountered when using the Internet is probably the World Wide Web (WWW). The web is also called WWW, using only the first letters of the English words, or simply Web. The web symbolizes a web, meaning a spider web.

Because the web is implemented graphically, one of its greatest advantages is convenient information search. It can display text and graphics on the screen at the same time, and thanks to its excellent user interface, text, graphics, sound, video, and similar content can be handled easily in a browser.

Web Browser

A web browser is a client program that allows users to view hypertext documents on a web server using the Internet. It can process not only text but also multimedia data through hypermedia functions. When users want audio or video information, it can load and execute the required programs to handle various resources on the web.

Basic functions provided by a web browser include opening web pages, providing a list of recently visited URLs (Uniform Resource Locators), remembering and managing frequently visited URLs, and saving and printing web pages. Without a web browser, even if you are connected to the Internet, you cannot view homepages on the Internet.

Types of Web Browsers

Google Chrome.
Mozilla Firefox.
Chromium Microsoft Edge.
Opera browser
Vivaldi browser
Naver Whale
Brave browser

How the Web Works

Image source: http://tcpschool.com/webbasic/works

The user enters a URL in the browser.
The browser finds the server’s IP address through DNS.
It creates an HTTP request message using the HTTP protocol.
It sends the HTTP request message through a TCP/IP connection.
The server creates an HTTP response message using the HTTP protocol.
It sends the response to the requesting computer through a TCP/IP connection.
The arrived HTTP response message is returned as web page data and rendered by the web browser so the user can view it.

International Standards for Web Accessibility

Web standards designed to improve web accessibility refer to commonly used technologies and rules.
The W3C (World Wide Web Consortium) was established to define web standards.
Web standards define HTML as the structure language, CSS as the presentation language, and Script as the behavior language.

Difference Between Forward and Redirect

kc@example.com (kc kim) — Thu, 07 Apr 2022 01:19:00 +0900

Forward Method

Forward only moves pages at the Web Container level. In practice, the web browser cannot tell that it has moved to another page. Therefore, the web browser displays the URL that was first called, and the URL information of the page that was moved to cannot be checked.
The currently running page and the page called by forward share the Request object and Response object.

Statement used when moving from a Servlet to a JSP page

RequestDispatcher rd = request.getRequestDispatcher("test01.jsp");
rd.forward(request, response);

Redirect Method

When a Redirect command enters the Web Container, it instructs the web browser to move to another page. Then the web browser changes the URL to the instructed address and moves to that address. It moves to an address in another web container.
On the new page, new Request and Response objects are created.

Statement used when moving from a Servlet to a JSP page

response.sendRedirect("test02.jsp");

Source

Difference between redirect and forward