![]() The value can be any regular expression matched against the query parameter name. That the query parameter location, which comes from the crawling root, also applies to the child pages, the option "Inherit Crawl Root Query Parameter Pattern" must be set to the value location. Likewise, child pages provide different content: and. For example, The following crawling roots and provide different content. ![]() The use case is, for example, web pages that provide different content depending on the query parameters. The “Inherit Crawling Root Query Parameter Pattern” option lets you inherit URL query parameters from the crawling root to the children's URLs. If this field is left empty and " Enable Default ACLs" is active, "everyone" is used by default as " Default ACL Principals". Several " Default ACL Principals" can be specified separated by line breaks. Which ACLs are set in these cases is determined with the option " Default ACL Principals". If the " Enable Default ACLs" option is active (active by default), ACLs are set for Web documents if they do not have any explicitly defined ACLs (e.g. If a regular expression is set as an “Enforce extension from URL if matches” parameter, the extension is derived from the URL instead of from the “Content type” http header for documents with matching URLs. For this see section “Sitemap Crawling Strategy”. ![]() IMPORTANT: the option “Incomplete Delta Crawl Runs” must not be used with sitemap-based delta crawling. To minimize the load of subsequent crawl runs on your site, you can provide a crawling root with links to updated pages only. With the option “Incomplete Delta Crawl Runs” enabled, pages that are not reachable from the current “Crawling Root” are not deleted from the index at the end of the crawl run. for setting the Accept-Language header), do so using the Accept Headers parameter. If you want to set additional HTTP headers while crawling (e.g. URLs having a higher hop count will be ignored. With the “Maximum Link Depth” field you can set the maximum count of hops from the crawling roots for the URLs that are crawled. You can remove existing crawling roots by clicking on the “Remove” button besides them. The added crawling roots are displayed in the list above (e.g. You can add an arbitrary number of crawling roots editing the “Crawling Root” field and pressing the “Add” button. With the option “Convert Document Keys to Lower Case” set, the document keys (header/mes:key) of the documents are converted to lower case. If "URL Regex", "URL Exclude Pattern" and "Include/Exclude URL by Metadata" are used simultaneously, "URL Regex" is applied first, then the pages are excluded with "URL Exclude Pattern" and finally the remaining pages are filtered with "Include/Exclude URL by Metadata". The Metadata Name field specifies the metadata name and the Pattern field specifies the regular expression against which the metadata value is matched. With the option "Include URL By Metadata" or "Exclude URL by Metadata", certain pages can be excluded (when crawling sitemaps) based on the metadata in the sitemap. The pattern has to match the whole URL (including URL parameters). The URLs matched by this pattern will not be crawled and hence not be used for further link extraction. You can also specify a pattern for the URLs that need to be excluded using the “URL Exclude Pattern” field. If you leave the field empty, all pages with the same host and domain parts as the “Crawling Root” will be indexed (e.g. ![]() You can specify a regular expression for the links to follow with the field “URL Regex”. With the “Crawler Interval” setting you are able to configure the interval between two crawl runs. ![]() If necessary, choose “Web” in the “Category” field. Adapt the Display Name of the Index Service and the related Filter Service if necessaryĪdd a new data source with the symbol “Add new custom source” at the bottom right. This video describes how to configure a basic Web Connector to index a website for both with and without a sitemap: Configuration of Mindbreeze Configuration of Index and CrawlerĬlick on the “ Indices” tab and then on the “ Add new index” symbol to create a new index.Įnter the index path, e.g. Video Tutorial „Set up a basic Web Connector” The term ‘user‘ is used in a gender-neutral sense throughout the document. No rights to our software or our professional services, or results of our professional services, or other protected rights can be based on the handing over and presentation of these documents.ĭistribution, publication or duplication is not permitted. All hardware and software names used are registered trade names and/or registered trademarks of the respective manufacturers. ![]()
0 Comments
Leave a Reply. |
Details
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |