Why does to_tsvector ignore HTML script tags?

Why does the to_tsvector function in PostgreSQL ignore HTML script tags, and how does it handle or filter out HTML content during text search?

The to_tsvector function in PostgreSQL is designed to parse and tokenize text for full-text search. It ignores HTML script tags (<script> and <style>) because these tags typically contain code or styling information that is not relevant to text search. Here's how it works:

HTML Tag Handling: PostgreSQL's text search parser is hardcoded to recognize and ignore specific HTML tags, such as <script> and <style>. This means that any content within these tags is excluded from the resulting token vector.

Tokenization Process: When to_tsvector processes text, it breaks the text into tokens and assigns types to each token. Tags like <script> and <style> are identified, and their contents are ignored.

Parser Configuration: The parser configuration determines how different types of tokens are handled. By default, tags like <script> and <style> are treated as whitespace, resulting in an empty token vector for their contents.