Indexes¶
This page will try to detail each configuration available into .PyFunceble.yaml
along with the location of where we are looking for the configuration file.
adblock
¶
Type:
boolean
Default value:
False
Description: Enable / disable the adblock format decoding.
Note
If this index is set to True
, every time we read a given file, we try to extract the elements that are present.
We basically only decode the adblock format.
Note
If this index is set to False
, every time we read a given file, we will consider one line as an element to test.
auto_continue
¶
Type:
boolean
Default value:
True
Description: Enable / disable the auto continue system.
command
¶
Type:
string
Default value:
""
Description: Set the command to run before each commit (except the final one).
Note
The parsed command is called only if auto_continue
and travis
are set to True
.
command_before_end
¶
Type:
string
Default value:
""
Description: Set the command to run before the final commit.
Note
The parsed command is called only if auto_continue
and travis
are set to True
.
Note
Understand by final commit the commit which will deliver the last element we have to test.
custom_ip
¶
Type:
string
Default value:
"0.0.0.0"
Description: Set the custom IP to use when we generate a line in the hosts file format.
Note
This index has no effect if generate_hosts
is set to False
.
days_between_db_retest
¶
Type:
integer
Default value:
1
Description: Set the number of day(s) between each retest of the
INACTIVE
andINVALID
elements which are present intoinactive_db.json
.
Note
This index has no effect if inactive_database
is set to False
.
db_type
¶
Type:
string
Default value:
json
Available values:
json
,mariadb
,mysql
,sqlite
Description: Set the database type to use everytime we create a database.
Note
This feature is applied to the following subsystems:
- Autocontinue physically located (JSON) at
output/continue.json
. - InactiveDB physically located (JSON) at
[config_dir]/inactive_db.json
. - Mining physically located (JSON) at
[config_dir]/mining.json
. - WhoisDB physically located (JSON) at
[config_dir]/whois.json
.
debug
¶
Type:
boolean
Default value:
False
Description: Enable / disable the generation of debug file(s).
Note
This index has no effect if logs
is set to False
Warning
Do not touch this index unless you a have good reason to.
Warning
Do not touch this index unless you have been invited to.
dns_server
¶
Type:
None
orlist
Default value:
null
Description: Set the DNS server(s) to work with.
Note
When a list is given the following format is expected.
dns_server:
- dns1.example.org
- dns2.example.org
Warning
We expect DNS server(s). If a non-DNS server is given. You’ll get almost all results
as INACTIVE
.
filter
¶
Type:
string
Default value:
""
Description: Set the element to filter.
Note
This index should be initiated with a regular expression.
generate_complements
¶
Type:
boolean
Default value:
False
Description: Enable / disable the generation and test of the complements.
Note
A complement is for example example.org
if www.example.org
is given and vice-versa.
generate_hosts
¶
Type:
boolean
Default value:
True
Description: Enable / disable the generation of the hosts file(s).
generate_json
¶
Type:
boolean
Default value:
False
Description: Enable / disable the generation of the JSON file(s).
header_printed
¶
Type:
boolean
Default value:
False
Description: Say to the system if the header has been already printed or not.
Warning
Do not touch this index unless you have a good reason to.
hierarchical_sorting
¶
Type:
boolean
Default value:
False
Description: Say to the system if we have to sort the list and the outputs in a hierarchical order.
iana_whois_server
¶
Type:
string
Default value:
whois.iana.org
Description: Set the server to call to get the
whois
referer of a given element.
Note
This index is only used when generating the iana-domains-db.json
file.
Warning
Do not touch this index unless you a have good reason to.
idna_conversion
¶
Type:
boolean
Default value:
False
Description: Tell the system to convert all domains to IDNA before testing.
Note
We use domain2idna for the conversion.
inactive_database
¶
Type:
boolean
Default value:
True
Description: Enable / Disable the usage of a database to store the
INACTIVE
andINVALID
element to retest overtime.
less
¶
Type:
boolean
Default value:
True
Description: Enable / Disable the output of every information of screen.
local
¶
Type:
boolean
Default value:
False
Description: Enable / Disable the execution of the test(s) in a local or private network.
logs
¶
Type:
boolean
Default value:
True
Description: Enable / Disable the output of all logs.
maximal_processes
¶
Type:
integer
Default value:
25
Description: Set the number of maximal simultaneous processes to use/create/run.
mining
¶
Type:
boolean
Default value:
True
Description: Enable / Disable the mining subsystem.
multiprocess
¶
Type:
boolean
Default value:
False
Description: Enable / Disable the usage of multiple processes instead of the default single process.
no_files
¶
Type:
boolean
Default value:
False
Description: Enable / Disable the generation of any file(s).
no_special
¶
Type:
boolean
Default value:
False
Description: Enable / Disable the usage of the SPECIAL rules - which are discribes in the source column section.
no_whois
¶
Type:
boolean
Default value:
False
Description: Enable / Disable the usage of
whois
in the tests.
plain_list_domain
¶
Type:
boolean
Default value:
False
Description: Enable / Disable the generation of the plain list of elements sorted by statuses.
Warning
Do not touch this index unless you a have good reason to.
quiet
¶
Type:
boolean
Default value:
False
Description: Enable / Disable the generation of output on the screen.
referer
¶
Type:
string
Default value:
""
Description: Set the referer of the element that is currently under test.
Warning
Do not touch this index unless you a have good reason to.
seconds_before_http_timeout
¶
Type:
integer
Default value:
3
Description: Set the timeout to apply to every HTTP status code request.
Note
This index must be a multiple of 3
.
show_execution_time
¶
Type:
boolean
Default value:
False
Description: Enable / disable the output of the execution time.
show_percentage
¶
Type:
boolean
Default value:
True
Description: Enable / disable the output of the percentage of each status.
simple
¶
Type:
boolean
Default value:
False
Description: Enable / disable the simple output mode.
Note
If this index is set to True
, the system will only return the result inf format: tested.element STATUS
.
split
¶
Type:
boolean
Default value:
True
Description: Enable / disable the split of the results files.
Note
Understand with “results files” the mirror of what is shown on screen.
syntax
¶
Type:
boolean
Default value:
False
Description: Enable / disable the syntax (only) testing.
Warning
If this index is set to True
, we ONLY check for syntax, not availability.
travis
¶
Type:
boolean
Default value:
False
Description: Enable / disable the Travis CI autosaving system.
Warning
Do not activate this index unless you are using PyFunceble under Travis CI.
travis_autosave_commit
¶
Type:
string
Default value:
"PyFunceble - AutoSave"
Description: Set the default commit message we want to use when have to commit (save) but our tests are not yet completed.
travis_autosave_final_commit
¶
Type:
string
Default value:
"PyFunceble - Results"
Description: Set the default final commit message we want to use when we all tests are finished.
travis_autosave_minutes
¶
Type:
integer
Default value:
15
Description: Set the minimum of minutes we have to run before to automatically save our test results.
Note
As many services are setting a rate limit per IP, it’s a good idea to set this value between 1
and 15
minutes.
travis_branch
¶
Type:
string
Default value:
master
Description: Set the git branch where we are going to push our results.
unified
¶
Type:
boolean
Default value:
False
Description: Enable / Disable the generation of the unified results.
Note
This index has no effect if split
is set to True
.
user_agent
¶
Type:
string
Default value:
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"
Description: Set the User-Agent to use every time we are requesting something from a web server other than our API.
verify_ssl_certificate
¶
Type:
boolean
Default value:
False
Description: Enable / Disable the verification of the SSL/TLS certificate when testing for URL.
Warning
If you set this index to True
, you may get false positive result.
Indeed if the certificate is not registered to the CA or is simply invalid and the domain is still alive, you will always get INACTIVE
as output.
whois_database
¶
Type:
boolean
Default value:
True
Description: Enable / Disable the usage of the whois database to avoid/bypass whois server requests rate limit.
outputs
¶
Type:
dict
Description: Set the needed output tree/names.
Warning
If you choose to change anything please consider deleting our output/
directory and the dir_structure*.json
files.
outputs[default_files]
¶
Type:
dict
Description: Set the default name of some important files.
outputs[default_files][dir_structure]
¶
Type:
string
Default value:
dir_structure.json
Description: Set the default filename of the file which has the structure to re-construct.
Note
This index has no influence with dir_structure_production.json
outputs[default_files][iana]
¶
Type:
string
Default value:
iana-domains-db.json
Description: Set the default filename of the file which has the formatted copy of the IANA root zone database.
outputs[default_files][inactive_db]
¶
Type:
string
Default value:
inactive_db.json
Description: Set the default filename of the file which will save the list of elements to retest overtime.
outputs[default_files][results]
¶
Type:
string
Default value:
results.txt
Description: Set the default filename of the file which will save the formatted copy of the public suffix database.
outputs[default_files][public_suffix]
¶
Type:
string
Default value:
public-suffix.json
Description: Set the default filename of the file which will save the mirror of what is shown on screen.
outputs[default_files][mining]
¶
Type:
string
Default value:
mining.json
Description: Set the default filename of the file which will save the temporary list of mined subject to test.
outputs[default_files][whois_db
¶
Type:
string
Default value:
whois_db.json
Description: Set the default filename of the file which will save the whois information for caching.
outputs[default_files][sqlite]
¶
Type:
string
Default value:
pyfunceble.db
Description: Set the default filename of the file which will save all database in sqlite format.
outputs[db_type]
¶
Type:
dict
Description: Set the default name of some important files related to the
db_type
index.
outputs[db_type][directory]
¶
Type:
string
Default value:
.db_type/
Description: Set the default directory where we have to save the SQL files which we are later going to be used for database generation/creation.
outputs[db_type][files]
¶
Type:
dict
Description: Set the default name of some important files related to the
db_type
index.
outputs[db_type][files][sqlite]
¶
Type:
string
Default value:
sqlite.sql
Description: Set the default name of the file to call when we have to construct the SQLite database.
outputs[domains]
¶
Type:
dict
Description: Set the default name of some important files related to the
plain_list_domain
index.
outputs[domains][directory]
¶
Type:
string
Default value:
domains/
Description: Set the default directory where we have to save the plain list of elements for each status.
outputs[domains][filename]
¶
Type:
string
Default value:
list
Description: Set the default filename of the file which will save the plain list of elements.
outputs[hosts]
¶
Type:dict
Description: Set the default name of some important files related to the
generate_hosts
index.
outputs[hosts][directory]
¶
Type:
string
Default value:
hosts/
Description: Set the default directory where we have to save the hosts files of the elements for each status.
outputs[hosts][filename]
¶
Type:
string
Default value:
hosts
Description: Set the default filename of the file which will save the hosts files of the elements.
outputs[json]
¶
Type:dict
Description: Set the default name of some important files related to the
generate_json
index.
outputs[json][directory]
¶
Type:
string
Default value:
json
Description: Set the default directory where we have to save the JSON files of the elements for each status.
outputs[json][filename]
¶
Type:
string
Default value:
dump.json
Description: Set the default filename of the file which will save the JSON files of the elements.
outputs[complements]
¶
Type:dict
Description: Set the default name of some important files/directories related to the
generate_complements
index.
outputs[complements][directory]
¶
Type:
string
Default value:
complements
Description: Set the default directory where we have to save the complements related files sorted by status.
outputs[analytic]
¶
Type:dict
Description: Set the default name of some important files and directories related to the
generate_hosts
index.
outputs[analytic][directories]
¶
Type:
dict
Description: Set the default name of some important directories related to the
http_codes[active]
index.
outputs[analytic][directories][parent]
¶
Type:
string
Default value:
Analytic/
Description: Set the default directory where we are going to put everything related to the HTTP analytic.
outputs[analytic][directories][potentially_down]
¶
Type:
string
Default value:
POTENTIALLY_INACTIVE/
Description: Set the default directory where we are going to put all potentially inactive data.
outputs[analytic][directories][potentially_up]
¶
Type:
string
Default value:
POTENTIALLY_INACTIVE/
Description: Set the default directory where we are going to put all potentially active data.
outputs[analytic][directories][up]
¶
Type:
string
Default value:
POTENTIALLY_INACTIVE/
Description: Set the default directory where we are going to put all active data.
outputs[analytic][directories][suspicious]
¶
Type:
string
Default value:
SUSPICIOUS/
Description: Set the default directory where we are going to put all suspicious data.
outputs[analytic][filenames]
¶
Type:
dict
Description: Set the default name of some important files related to the
http_codes[active]
index and the HTTP analytic subsystem.
outputs[analytic][filenames][potentially_down]
¶
Type:
string
Default value:
down_or_potentially_down
Description: Set the default filename where we are going to put all potentially inactive data.
outputs[analytic][filenames][potentially_up]
¶
Type:
string
Default value:
potentially_up
Description: Set the default filename where we are going to put all potentially active data.
outputs[analytic][filenames][up]
¶
Type:
string
Default value:
active_and_merged_in_results
Description: Set the default filename where we are going to put all active data.
outputs[analytic][filenames][suspicious]
¶
Type:
string
Default value:
suspicious_and_merged_in_results
Description: Set the default filename where we are going to put all suspicious data.
outputs[logs]
¶
Type:
dict
Description: Set the default name of some important files and directories related to the
logs
index.
outputs[logs][directories]
¶
Type:dict
Description: Set the default name of some important directories related to the
logs
index.
outputs[logs][directories][date_format]
¶
Type:
string
Default value:
date_format/
Description: Set the default directory where we are going to put everything related to the data when the dates are in the wrong format.
outputs[logs][directories][no_referer]
¶
Type:
string
Default value:
no_referer/
Description: Set the default directory where we are going to put everything related to the data when no referer is found.
outputs[logs][directories][parent]
¶
Type:
string
Default value:
no_referer/
Description: Set the default directory where we are going to put everything related to the data when no referer is found.
outputs[logs][directories][percentage]
¶
Type:
string
Default value:
percentage/
Description: Set the default directory where we are going to put everything related to percentages.
outputs[logs][directories][whois]
¶
Type:
string
Default value:
whois/
Description: Set the default directory where we are going to put everything related to whois data.
Note
This is the location of all files when the debug
index is set to True
.
outputs[logs][filenames]
¶
Type:
dict
Description: Set the default filenames of some important files related to the
logs
index.
outputs[logs][filenames][auto_continue]
¶
Type:
string
Default value:
continue.json
Description: Set the default filename where we are going to put the data related to the auto continue subsystem.
Note
This file is allocated if the auto_continue
is set to True
.
outputs[logs][filenames][execution_time]
¶
Type:
string
Default value:
execution.log
Description: Set the default filename where we are going to put the data related to the execution time.
Note
This file is allocated if the show_execution_time
is set to True
.
outputs[logs][filenames][percentage]
¶
Type:
string
Default value:
percentage.txt
Description: Set the default filename where we are going to put the data related to the percentage.
Note
This file is allocated if the show_percentage
is set to True
.
outputs[main]
¶
Type:
string
Default value:
""
Description: Set the default location where we have to generate the
parent_directory
directory and its dependencies.
outputs[parent_directory]
¶
Type:
string
Default value:
output/
Description: Set the directory name of the parent directory which will contain all previously nouned directories.
status
¶
Type:
dict
Description: Set the needed, accepted and status name.
status[list]
¶
Type:
dict
Description: Set the needed and accepted status name.
Warning
All status should be in lowercase.
status[list][valid]
¶
Type:
list
Default value:
["valid","syntax_valid","valid_syntax"]
Description: Set the accepted
VALID
status.
Note
This status is only shown if the syntax
index is activated.
status[list][up]
¶
Type:
list
Default value:
["up","active"]
Description: Set the accepted
ACTIVE
status.
status[list][generic]
¶
Type:
list
Default value:
["generic"]
Description: Set the accepted
generic
status.
Note
This status is the one used to say the system that we have to print the complete information on the screen.
status[list][http_active]
¶
Type:
list
Default value:
["http_active"]
Description: Set the accepted status for the
outputs[analytic][filenames][up]
index.
status[list][down]
¶
Type:
list
Default value:
["down","inactive", "error"]
Description: Set the accepted status
INACTIVE
index.
status[list][invalid]
¶
Type:
list
Default value:
["ouch","invalid"]
Description: Set the accepted status
INVALID
index.
status[list][potentially_down]
¶
Type:
list
Default value:
["potentially_down", "potentially_inactive"]
Description: Set the accepted status for the
outputs[analytic][filenames][potentially_down]
index.
status[list][potentially_up]
¶
Type:
list
Default value:
["potentially_up", "potentially_active"]
Description: Set the accepted status for the
outputs[analytic][filenames][potentially_up]
index.
status[list][suspicious]
¶
Type:
list
Default value:
["strange", "hum", "suspicious"]
Description: Set the accepted status for the
outputs[analytic][filenames][suspicious]
index.
status[official]
¶
Type:
dict
Description: Set the official status name.
Note
Those status are the ones that are printed on the screen.
Warning
After any changes here please delete dir_structure.json
and the output/
directory.
status[official][up]
¶
Type:
string
Default value:
ACTIVE
Description: Set the returned status for the
ACTIVE
case.
status[official][down]
¶
Type:
string
Default value:
INACTIVE
Description: Set the returned status for the
INACTIVE
case.
status[official][invalid]
¶
Type:
string
Default value:
INVALID
Description: Set the returned status for the
INVALID
case.
status[official][valid]
¶
Type:
string
Default value:
VALID
Description: Set the returned status for the
VALID
case.
Note
This status is only shown if the syntax
index is activated.
http_codes
¶
Type:
dict
Description: Handle the interpretation of each status codes when we do and generate our analytic data.
http_codes[active]
¶
Type:
boolean
Default value:
True
Description: Enable / Disable the usage of the HTTP status code extraction.
http_codes[list]
¶
Type:
dict
Description: Categorize the http status code as mentioned in the documentation related to the
HTTP Code
column.
http_codes[list][up]
¶
Type:
list
- Default value:
- 100 - 101 - 200 - 201 - 202 - 203 - 204 - 205 - 206Description: List the HTTP status codes which are considered as
ACTIVE
.
http_codes[list][potentially_down]
¶
Type:
list
- Default value:
- 400 - 402 - 403 - 404 - 409 - 410 - 412 - 414 - 415 - 416Description: List the HTTP status code which are considered as
INACTIVE
orPOTENTIALLY_INACTIVE
.
http_codes[list][potentially_up]
¶
Type:
list
- Default value:
- 000 - 300 - 301 - 302 - 303 - 304 - 305 - 307 - 403 - 405 - 406 - 407 - 408 - 411 - 413 - 417 - 500 - 501 - 502 - 503 - 504 - 505Description: List the HTTP status code which are considered as
ACTIVE
orPOTENTIALLY_ACTIVE
.
links
¶
Type:
dict
Description: Set the list of links which can be used/called by the system when needed.
Note
The objective of this index is to avoid hardcoded links when the configuration file is readable.
links[api_date_format]
¶
Type:
string
Default value:
https://pyfunceble.funilrys.com/api/date-format
Description: Set the link to use when we share logs.
links[api_no_referer]
¶
Type:
string
Default value:
https://pyfunceble.funilrys.com/api/no-referer
Description: Set the link to use when we share logs.
links[config]
¶
Type:
string
Default value:
https://raw.githubusercontent.com/funilrys/PyFunceble/master/.PyFunceble_production.yaml
Description: Set the upstream link to the configuration file.
links[dir_structure]
¶
Type:
string
Default value:
https://raw.githubusercontent.com/funilrys/PyFunceble/master/dir_structure_production.json
Description: Set the upstream link to the directory structure dump file.
links[iana]
¶
Type:
string
Default value:
https://raw.githubusercontent.com/funilrys/PyFunceble/master/iana-domains-db.json
Description: Set the upstream link to the IANA zone file configuration file.
links[mariadb]
¶
Type:
string
Default value:
https://raw.githubusercontent.com/funilrys/PyFunceble/master/db_types/mariadb.sql
Description: Set the upstream link to the MariaDB table constructor file.
links[mysql]
¶
Type:
string
Default value:
https://raw.githubusercontent.com/funilrys/PyFunceble/master/db_types/mysql.sql
Description: Set the upstream link to the MySQL table constructor file.
links[psl]
¶
Type:
string
Default value:
https://raw.githubusercontent.com/funilrys/PyFunceble/master/public-suffix.json
Description: Set the upstream link to the public suffix database file.
links[repo]
¶
Type:
string
Default value:
https://github.com/funilrys/PyFunceble
Description: Set the upstream link to the repository.
links[requirements]
¶
Type:
string
Default value:
https://raw.githubusercontent.com/funilrys/PyFunceble/master/requirements.txt
Description: Set the upstream link to the
requirements.txt
file.
links[sqlite]
¶
Type:
string
Default value:
https://raw.githubusercontent.com/funilrys/PyFunceble/master/sqlite.sql
Description: Set the upstream link to the SQLite table constructor file.