131
CONFIGURING AND ADMINISTERING COLDFUSION 9
Indexing Collections with Verity Spider
Last updated 2/21/2012
-noflowctrl
Type
Web crawling only
Disables round-robin indexing of websites with network flow control.
By default, Verity Spider uses round-robin indexing of websites to avoid overwhelming a web server and to improve
indexing performance. Verity Spider connects to each web server in a round-robin manner, using up to the value for
the
-connections option. This means that one URL is fetched from each web server, in turn.
Note: Using the -noflowctrl option can result in a significant drop in performance.
-noproxy
Type
Web crawling only
Syntax
-noproxy name_1 [name_n] ...
Used with the -proxy option, the -noproxy option specifies that Verity Spider directly access the hosts whose names
match those specified. By default, when you specify the
-proxy option, Verity Spider first tries to access every host
with the proxy information. To improve performance, use the
-noproxy option for the hosts you know can be
accessed without a proxy host. For the name variable, you can use the asterisk (*) wildcard for text strings; for example:
'*.verity.com'
You cannot use the question mark (?) wildcard, and the -regexp option does not let you use regular expressions.
In Windows, include double-quotation marks around the argument to protect the asterisk special character (*). On
UNIX, use single-quotation marks. This is only required when you run the indexing job from a command line.
Quotation marks are not necessary within a command file (the
-cmdfile option).
Note: You must have valid Verity Spider licensing capability to use this option.
-proxy
Type
Web crawling only
Syntax
-proxy proxyhost:port
Specifies host and port for proxy server.
Note: You must have valid Verity Spider licensing capability to use this option.
See also “-proxyauth” on page 132 for proxy servers that require authentication, and “-noproxy” on page 131 for hosts
that you know are accessible without having to go through a proxy server.
Comments to this Manuals