Configuration¶
Location¶
Problematics¶
- How can we create a more efficient way to work with configuration?
- How can we make the configuration file(s) available globally so that PyFunceble can be run everywhere in the user workspace?
To answer those problematics, we moved the configuration location elsewhere in the place where most users expect to have their configuration file(s).
Filename-s¶
At any of the listed configuration location, the following file will be read:
.PyFunceble.yaml
This file is generated automatically if missing.
This file is automatically replaced if you choose to merge the upstream configuration when a features key is introduced or removed.
This “new” behavior was introduced at the same time .PyFunceble.overwrite.yaml was. This is to ensure PyFunceble at all time is running on a fully compatible version of .PyFunceble.yaml
.PyFunceble.overwrite.yaml
- This is the project specific configuration file for PyFunceble.
This file is now taking presedence over
.PyFunceble.yaml
which for sure previously was the file you used to alter for your pwn needs. - This file can only be maintained by a human-controlled process. PyFunceble will never write into and never owerwrite this file, once it is created.
- This is the project specific configuration file for PyFunceble.
This file is now taking presedence over
Repository clone¶
If you cloned the repository and you’re trying to test from a cloned
directory (the one with for example CONTRIBUTING.md
) we consider
the configuration directory as the current one.
Note
This behavior allows us to not modify the way we develop PyFunceble.
Travis CI¶
Under Travis CI, we search or initiate the configuration at the directory we are currently located.
Warning
We don’t care about the distribution, as long as the
TRAVIS_BUILD_DIR
environment variable is set, we search or
initiate the configuration in the current directory.
Note
If you want to force the directory where we should work, you can
initiate the PYFUNCEBLE_CONFIG_DIR
environment variable with
the path where we should work.
GitLab CI/CD¶
Under GitLab CI/CD, we search or initiate the configuration at the directory we are currently located.
Warning
We don’t care about the distribution, as long as the PROJECT_CI
and GITLAB_CI
environment variables are set, we search or
initiate the configuration in the current directory.
Note
If you want to force the directory where we should work, you can
initiate the PYFUNCEBLE_CONFIG_DIR
environment variable with
the path where we should work.
GitHub Actions¶
Under GitHub Actions, we search or initiate the configuration at the directory we are currently located.
Warning
We don’t care about the distribution, as long as the
GITHUB_ACTIONS
environment variable is set, we search or
initiate the configuration in the current directory.
Note
If you want to force the directory where we should work, you can
initiate the PYFUNCEBLE_CONFIG_DIR
environment variable with
the path where we should work.
Jenkins CI¶
Under Jenkins CI, we search or initiate the configuration at the directory we are currently located.
Warning
We don’t care about the distribution, as long as the
JENKINS_URL
and JENKINS_HOME
environment variables
are set, we search or initiate the configuration in the current
directory.
Note
If you want to force the directory where we should work, you can
initiate the PYFUNCEBLE_CONFIG_DIR
environment variable with
the path where we should work.
Linux and MacOS (Darwin Kernel)¶
Under Linux and MacOS, we look for the following directories in their order. If any configuration directory is found, the system proposes you to install them automatically on the first configuration file.
~/.config/PyFunceble
~/.PyFunceble
${PWD}
Note
If the parent directory does not exist, we move to the next possible location in the given order.
This means that under most Linux distributions and MacOS versions,
we consider ~/.config/PyFunceble
as the configuration
location.
But if the ~/.config
directory does not exist, we fallback
to ~/.PyFunceble
as the configuration location.
Windows¶
As mentioned by Pat Altimore’s Blog, we used the
Per user configuration files synchronized across domain joined
machines via Active Directory Roaming
section to understand what we
should do to find our configuration directory.
Under Windows, we look for the following directories in their order. If any configuration directory is found, the system proposes you to install them automatically on the first configuration file.
%APPDATA%\PyFunceble
(environnement variable)%CD%
Note
%CD%
is explained by the set command (set /?
):
%CD% - expands to the current directory string.
Note
If the parent directory does not exist, we move to the next possible location in the given order.
This means that under most Windows versions, we consider
%APPDATA%\PyFunceble
- also know as
C:\Users\userName\AppData\Roaming\PyFunceble
- as the
configuration location.
But if the %APPDATA%
directory does not exist, we fall back
to the current directory as the configuration location.
Custom location¶
Sometimes, you may find yourself in a position where you absolutely do not want PyFunceble to use its default configuration location.
For that reason, if you set your desired configuration location along
with the PYFUNCEBLE_CONFIG_DIR
environment variable, we take
that location as the (default) configuration location.
Autoconfiguration¶
Sometimes, you may find yourself in a position that you do not or you can’t answer the question which asks you if you would like to merge an upstream configuration file into your local one.
For that reason, if you can set the PYFUNCEBLE_AUTO_CONFIGURATION
as an environment variable. Setting that environment variable will make
PyFunceble merge the upstream configuration when a new key is available but
not found into your configuration file.
Warning
As of 4.0.0
the configuration file is automatically copied into
your configuration directory - if it is not found.
Indexes¶
This page will try to detail each configuration available into .PyFunceble.yaml
along with the location of where we are looking for the configuration file.
verify_ssl_certificate
¶
Type:
boolean
Default value:
False
Description: Activates or disables the verification of the SSL/TLS certificate when testing for URL.
Warning
If you set this index to True
, you may get false positive
result.
Indeed if the certificate is not registered to the CA or is simply
invalid and the domain is still alive, you will always get
INACTIVE
as output.
debug
¶
Type:
dict
Description: Configures the debug / logging.
debug[active]
¶
Type:
boolean
Default value:
False
Description: Activate or deactivates the debug / logging.
debug[level]
¶
Type:
string
Default value:
info
Available values:
INFO
,DEBUG
,WARNING
,ERROR
,CRITICAL
.Description: Activate or deactivates the debug / logging.
cli_decoding
¶
Type:
dict
Description: Configures everything related to the decoding of the given input sources.
cli_decoding[adblock]
¶
Type:
boolean
Default value:
False
Description: Activates or disables the adblock decoding.
Note
If this index is set to True
, every time we read a given file, we
try to extract the elements that are present.
We basically only decode the adblock format.
Note
If this index is set to False
, every time we read a given file, we
will consider one line as an element to test.
cli_decoding[adblock_aggressive]
¶
Type:
boolean
Default value:
False
Description: Activates or disables disable some aggressive settings related to the adblock decoding.
Warning
This option is available but please keep in mind that the underlying source code is still experimental.
cli_decoding[rpz]
¶
Type:
boolean
Default value:
False
Description: Activates or disables the decoding of RPZ policies from each given input files.
cli_decoding[wildcard]
¶
Type:
boolean
Default value:
False
Description: Activates or disables the decoding of wildcards for each given input files.
cli_testing
¶
Type:
dict
Description: Configures everything related to the CLI testing.
cli_testing[hosts_ip]
¶
Type:
string
Default value:
"0.0.0.0"
Description: Sets the IP to prefix each lines of the hosts file.
cli_testing[max_workers]
¶
Type:
integer
Default value:
null
Description: Sets the number of maximal processes workers that we are allowed to allocate for the testing.
Warning
If set to null
, we use the default value calculated from your
machine ressources. Meaning:
CPU cores - 2
cli_testing[autocontinue]
¶
Type:
boolean
Default value:
True
Description: Activates or disables the automatic continuation subsystem.
cli_testing[inactive_db]
¶
Type:
boolean
Default value:
True
Description: Activates or disables the usage of a “database” to store all
INVALID
andINACTIVE
subject for continuous retest.
cli_testing[whois_db]
¶
Type:
boolean
Default value:
True
Description: Activates or disables the uage of a “database” to store the expiration date of all domains with a valid expiration date.
Warning
We do not recomend you to disable this. In fact, this is your safety against the rate limite imposed by most WHOIS servers.
cli_testing[cidr_expand]
¶
Type:
boolean
Default value:
False
Description: Activates or disables the expansion of CIDR formatted addresses.
cli_testing[complements]
¶
Type:
boolean
Default value:
False
Description: Activate or disables the generation and test of the complements of a given subject.
Note
A complement is for example example.org
if
www.example.org
is given and vice-versa.
cli_testing[cooldown_time]
¶
Type::
float
Default value:
0.0
Description: Sets the cooldown time to apply between each test.
cli_testing[db_type]
¶
Type:
string
Default value:
csv
Available values:
csv
,mariadb
,mysql
.Description: Sets the database type (or engine) to use everytime we create a database or a storage of a potentially huge dataset.
cli_testing[file_filter]
¶
Type:
string
Default value:
null
Description: A regular expression which we use to filter the subjects to (actually) test.
cli_testing[mining]
¶
Type:
boolean
Default value:
True
Description: Activates or disables the mining subsystem.
cli_testing[local_network]
¶
Type:
boolean
Default value:
False
Description: Activates or disables the consideration of the test(s) in or for a local or private network context.
cli_testing[preload_file]
¶
Type:
boolean
Default value:
False
Description: Activates or disables the preloading of the given input files. When this is activates, we preload the given files into the auto continue subsystem dataset in order to optimize some of our processes regarding the auto continue.
Note
This option does not have any effect if the auto continue subsystem is disabled.
cli_testing[ci]
¶
Type:
dict
Description: Configures everything related to the Continuous Integration.
cli_testing[ci][active]
¶
Type:
boolean
Default value:
False
Description: Activates or disables the Continuous Integration mechanism.
cli_testing[ci][commit_message]
¶
Type:
string
Default value:
"PyFunceble - AutoSave"
Description: Sets the commit message to apply everytime we have to apply a commit except for the really last one.
cli_testing[ci][end_commit_message]
¶
Type:
string
Default value:
"PyFunceble - Results"
Description: Sets the commit message to apply at the really end.
cli_testing[ci][max_exec_minutes]
¶
Type:
integer
Default value:
15
Description: Sets the number of minutes to wait before starting to stop a CI session.
Note
As many services are setting a rate limit per IP, it’s a good idea to set
this value between 1
and 15
minute(s).
cli_testing[ci][branch]
¶
Type:
string
Default value:
master
Description: Sets our git working branch. This is the branch from where we are supposed to store the tests (excepts the final results).
cli_testing[ci][distribution_branch]
¶
Type:
string
Default value:
master
Description: Sets our git distributions branch. This is the branch from where we are supposed to store and push the final results.
cli_testing[ci][command]
¶
Type:
string
Default value:
null
Description: Sets the command to execute before each commit (except the final one).
cli_testing[ci][end_command]
¶
Type:
string
Default value:
null
Description: Sets the command to execute before the final commit.
cli_testing[display_mode]
¶
Type:
dict
Description: Configures everything related to what is displayed.
cli_testing[display_mode][dots]
¶
Type:
boolean
Default value:
False
Description: Activate or disables the printing of dots or other characters when we skip the test of a subjec.
cli_testing[display_mode][dots]
¶
Type:
boolean
Default value:
False
Description: Activate or disables the display of dots or other characters when we skip the test of a subjec.
cli_testing[display_mode][execution_time]
¶
Type:
boolean
Default value:
False
Description: Activates or disables the display of the execution time.
cli_testing[display_mode][percentage]
¶
Type:
boolean
Default value:
True
Description: Activates or disables the display and generation of the percentage - file - of each status.
cli_testing[display_mode][quiet]
¶
Type:
boolean
Default value:
False
Description: Activates or disables the display of output to the terminal.
Warning
If the the dots mode is activate, this option will still allow them to work.
cli_testing[display_mode][less]
¶
Type:
boolean
Default value:
True
Description: Activates or disables the display of the minimal information in the table we print to stdout.
cli_testing[display_mode][all]
¶
Type:
boolean
Default value:
True
Description: Activates or disables the disply of the all information in the table we print to stdout.
cli_testing[display_mode][simple]
¶
Type:
boolean
Default value:
False
Description: Activates or disables the simple output mode.
Note
When this mode is active, the system will only return the result in the
following format: example.org ACTIVE
.
cli_testing[display_mode][status]
¶
Type:
string
|list
Default value:
all
Available values:
all
,ACTIVE
,INACTIVE
,INVALID
,VALID
,SANE
,MALICIOUS
Description: Sets the status that we are allowed to print to STDOUT.
Note
A list of status can be given if you want to filter multiple status at once.
cli_testing[display_mode][colour]
¶
Type:
boolean
Default value:
True
Description: Activates or disables the coloration to STDOUT.
cli_testing[testing_mode]
¶
Type:
dict
Description: Configures the testing mode to apply.
Warning
Only one of those is take in consideration.
Here is the priority / checking order:
syntax
reputation
availability
cli_testing[testing_mode][availability]
¶
Type:
boolean
Default value:
True
Description: Activates or disables the availability checker.
Note
This is the default mode.
cli_testing[testing_mode][syntax]
¶
Type:
boolean
Default value:
True
Description: Activates or disables the syntax checker.
cli_testing[testing_mode][reputation]
¶
Type:
boolean
Default value:
True
Description: Activates or disables the reputation checker.
cli_testing[days_between]
¶
Type:
dict
Description: Configures some days related events.
cli_testing[days_between][db_clean]
¶
Type:
integer
Default value:
28
Description: Sets the numbers of days since the introduction of a subject into the inactive dataset before it gets deleted.
Warning
As of PyFunceble 4.0.0
this is not actively implemented.
cli_testing[days_between][db_retest]
¶
Type:
integer
Default value:
28
Description: Sets the numbers of days since the introduction of a subject into the inactive dataset before it gets retested.
cli_testing[sorting_mode]
¶
Type:
dict
Description: Configures the sorting mode to apply.
Warning
Only one of those is take in consideration.
Here is the priority / checking order:
hierarchical
standard
cli_testing[sorting_mode][hierarchical]
¶
Type:
boolean
Default value:
False
Description: Activates or disables the sorting of the files content (output) in a hierarchical order.
cli_testing[sorting_mode][standard]
¶
Type:
boolean
Default value:
False
Description: Activates or disables the sorting of the files content (output) in our standard order.
cli_testing[file_generation]
¶
Type:
dict
Description: Configures everything related to the file generation.
cli_testing[file_generation][no_file]
¶
Type:
boolean
Default value:
False
Description: Activates or disables the generation of any non-logs file(s).
cli_testing[file_generation][no_file]
¶
Type:
boolean
Default value:
False
Description: Activates or disables the generation of any non-logs file(s).
cli_testing[file_generation][hosts]
¶
Type:
boolean
Default value:
True
Description: Activates or disables the generation of the hosts file(s).
cli_testing[file_generation][plain]
¶
Type:
boolean
Default value:
True
Description: Activates or disables the generation of the RAW file(s). What is meant is a list with only a list of subject (one per line).
cli_testing[file_generation][analytic]
¶
Type:
boolean
Default value:
True
Description: Activates or disables the generation of the analytic file(s).
cli_testing[file_generation][unified_results]
¶
Type:
boolean
Default value:
False
Description: Activates or disables the generation of the unified results file instead of the splitted one.
cli_testing[file_generation][merge_output_dirs]
¶
Type:
boolean
Default value:
False
Description: Activates or disables the merging of the outputs of all inputted files inside a single subdirectory as opposed to the normal behavior.
lookup
¶
Type:
dict
Description: Configures everything related to the lookup tools to use while testing a given subject.
lookup[dns]
¶
Type:
boolean
Default value:
True
Description: Activates or disables the usage of the DNS lookup whether possible.
lookup[http_status_code]
¶
Type:
boolean
Default value:
True
Description: Activates or disables the usage of the HTTP status code whether possible.
lookup[netinfo]
¶
Type:
boolean
Default value:
True
Description: Activates or disables the usage of the network information (or network socket) whether possible.
lookup[special]
¶
Type:
boolean
Default value:
True
Description: Activates or disables the usage of our SPECIAL and extra rules whether possible.
lookup[whois]
¶
Type:
boolean
Default value:
True
Description: Activates or disables the usage of the WHOIS record (or better said the expiration date in it) whether possible.
lookup[reputation]
¶
Type:
boolean
Default value:
True
Description: Activates or disables the usage of the reputation dataset whether possible.
lookup[timeout]
¶
Type:
integer
Default value:
5
Description: Sets the default timeout to apply to each lookup utilities everytime it is possible to define a timeout.
dns
¶
Type:
dict
Description: Configures everything related to the DNS lookup.
dns[server]
¶
Type:
list
Default value:
null
Description: Sets the DNS server(s) to work with.
Note
When a list is given the following format is expected.
dns_server:
- dns1.example.org
- dns2.example.org
Note
You can specify a port number to use to the DNS server if needed.
As example:
- 127.0.1.53:5353
Warning
Please be careful when you overwrite this option. If one is not correct,
you can almost sure that all results are going to be flagged as
INACTIVE
.
dns[protocol]
¶
Type:
string
Default value:
UDP
Available values:
UDP
,TCP
,HTTPS
,TLS
.Description: Sets the protocol to use for all DNS queries.
dns[follow_server_order]
¶
Type:
boolean
Default value:
True
Description: Activates or disables the follow-up of the given order.
dns[trust_server]
¶
Type:
boolean
Default value:
False
Description: Activates or disables the trust mode. When the trust mode is active and the first read DNS server gives us a negative response (without any error), we take it as it is.
Otherwise, when the trust mode is disabled, when the first read DNS server gives us a negative response (without any error), we still ask all other DNS servers that were given or found.
dns[delay]
¶
Type:
float
Default value:
0.0
Description: Sets the delay to apply between each DNS query.
user_agent
¶
Type:
dict
Description: Configures the user agent.
user_agent[browser]
¶
Type:
string
Default value:
chrome
Available values:
chrome
,edge
,firefox
,ie
,opera
,safari
.Description: Sets the browser to get the get the latest user agent from.
Warning
This option is not taken in consideration if user_agent[custom]
is
not set to null
.
user_agent[platform]
¶
Type:
string
Default value:
linux
Available values:
linux
,macosx
,win10
Description: Sets the platform to get the get the latest user agent for.
Warning
This option is not taken in consideration if user_agent[custom]
is
not set to null
.
user_agent[custom]
¶
Type:
string
Default value:
null
Description: Sets the user agent to use.
Warning
Setting this index will overwrite the choices made into
user_agent[platform]
and user_agent[browser]
.
http_codes
¶
Type:
dict
Description: Configures everything related to the HTTP status code and the way PyFunceble handles them.
http_codes[self_managed]
¶
Type:
bool
Default value:
False
Description: Informs PyFunceble that the status code list should not be managed automatically.
http_codes[list]
¶
Type:
dict
Description: Categorizes the HTTP status codes.
http_codes[list][up]
¶
Type:
list
Default value:
- 100 - 101 - 102 - 200 - 201 - 202 - 203 - 204 - 205 - 206 - 207 - 208 - 226Description: List the HTTP status codes which are considered as
ACTIVE
.
http_codes[list][potentially_down]
¶
Type:
list
Default value:
- 400 - 402 - 404 - 409 - 410 - 412 - 414 - 415 - 416 - 451Description: List the HTTP status code which are considered as
INACTIVE
orPOTENTIALLY_INACTIVE
.
http_codes[list][potentially_up]
¶
Type:
list
- Default value:
- 000 - 300 - 301 - 302 - 303 - 304 - 305 - 307 - 308 - 403 - 405 - 406 - 407 - 408 - 411 - 413 - 417 - 418 - 421 - 422 - 423 - 424 - 426 - 428 - 429 - 431 - 500 - 501 - 502 - 503 - 504 - 505 - 506 - 507 - 508 - 510 - 511Description: List the HTTP status code which are considered as
ACTIVE
orPOTENTIALLY_ACTIVE
.
links
¶
Type:
dict
Description: Sets the list of links which can be used/called by the system when needed.
links[api_date_format]
¶
Type:
string
Default value:
https://pyfunceble.funilrys.com/api/date-format
Description: Sets the link to use when we share logs.
links[api_no_referrer]
¶
Type:
string
Default value:
https://pyfunceble.funilrys.com/api/no-referrer
Description: Sets the link to use when we share logs.