Configuration

Location

Problematics

• How can we create a more efficient way to work with configuration?
• How can we make the configuration file(s) available globally so that PyFunceble can be run everywhere in the user workspace?

To answer those problematics, we moved the configuration location elsewhere in the place where most users expect to have their configuration file(s).

Filename-s

At any of the listed configuration location, the following file will be read:

• .PyFunceble.yaml

  • This file is generated automatically if missing.

  • This file is automatically replaced if you choose to merge the upstream configuration when a features key is introduced or removed.

    This “new” behavior was introduced at the same time .PyFunceble.overwrite.yaml was. This is to ensure PyFunceble at all time is running on a fully compatible version of .PyFunceble.yaml

• .PyFunceble.overwrite.yaml

  • This is the project specific configuration file for PyFunceble. This file is now taking presedence over .PyFunceble.yaml which for sure previously was the file you used to alter for your pwn needs.
  • This file can only be maintained by a human-controlled process. PyFunceble will never write into and never owerwrite this file, once it is created.

Repository clone

If you cloned the repository and you’re trying to test from a cloned directory (the one with for example CONTRIBUTING.md) we consider the configuration directory as the current one.

Note

This behavior allows us to not modify the way we develop PyFunceble.

Travis CI

Under Travis CI, we search or initiate the configuration at the directory we are currently located.

Warning

We don’t care about the distribution, as long as the TRAVIS_BUILD_DIR environment variable is set, we search or initiate the configuration in the current directory.

Note

If you want to force the directory where we should work, you can initiate the PYFUNCEBLE_CONFIG_DIR environment variable with the path where we should work.

GitLab CI/CD

Under GitLab CI/CD, we search or initiate the configuration at the directory we are currently located.

Warning

We don’t care about the distribution, as long as the PROJECT_CI and GITLAB_CI environment variables are set, we search or initiate the configuration in the current directory.

Note

If you want to force the directory where we should work, you can initiate the PYFUNCEBLE_CONFIG_DIR environment variable with the path where we should work.

GitHub Actions

Under GitHub Actions, we search or initiate the configuration at the directory we are currently located.

Warning

We don’t care about the distribution, as long as the GITHUB_ACTIONS environment variable is set, we search or initiate the configuration in the current directory.

Note

If you want to force the directory where we should work, you can initiate the PYFUNCEBLE_CONFIG_DIR environment variable with the path where we should work.

Jenkins CI

Under Jenkins CI, we search or initiate the configuration at the directory we are currently located.

Warning

We don’t care about the distribution, as long as the JENKINS_URL and JENKINS_HOME environment variables are set, we search or initiate the configuration in the current directory.

Note

If you want to force the directory where we should work, you can initiate the PYFUNCEBLE_CONFIG_DIR environment variable with the path where we should work.

Linux and MacOS (Darwin Kernel)

Under Linux and MacOS, we look for the following directories in their order. If any configuration directory is found, the system proposes you to install them automatically on the first configuration file.

1. ~/.config/PyFunceble
2. ~/.PyFunceble
3. ${PWD}

Note

If the parent directory does not exist, we move to the next possible location in the given order.

This means that under most Linux distributions and MacOS versions, we consider ~/.config/PyFunceble as the configuration location. But if the ~/.config directory does not exist, we fallback to ~/.PyFunceble as the configuration location.

Windows

As mentioned by Pat Altimore’s Blog, we used the Per user configuration files synchronized across domain joined machines via Active Directory Roaming section to understand what we should do to find our configuration directory.

Under Windows, we look for the following directories in their order. If any configuration directory is found, the system proposes you to install them automatically on the first configuration file.

1. %APPDATA%\PyFunceble (environnement variable)
2. %CD%

Note

%CD% is explained by the set command (set /?):

%CD% - expands to the current directory string.

Note

If the parent directory does not exist, we move to the next possible location in the given order.

This means that under most Windows versions, we consider %APPDATA%\PyFunceble - also know as C:\Users\userName\AppData\Roaming\PyFunceble- as the configuration location. But if the %APPDATA% directory does not exist, we fall back to the current directory as the configuration location.

Custom location

Sometimes, you may find yourself in a position where you absolutely do not want PyFunceble to use its default configuration location.

For that reason, if you set your desired configuration location along with the PYFUNCEBLE_CONFIG_DIR environment variable, we take that location as the (default) configuration location.

Autoconfiguration

Sometimes, you may find yourself in a position that you do not or you can’t answer the question which asks you if you would like to merge an upstream configuration file into your local one.

For that reason, if you can set the PYFUNCEBLE_AUTO_CONFIGURATION as an environment variable. Setting that environment variable will make PyFunceble merge the upstream configuration when a new key is available but not found into your configuration file.

Warning

As of 4.0.0 the configuration file is automatically copied into your configuration directory - if it is not found.

Indexes

This page will try to detail each configuration available into .PyFunceble.yaml along with the location of where we are looking for the configuration file.

share_logs

Type: boolean

Default value: False

Description: Activates or disables the logs sharing.

Warning

As the underlying API and infrastructure was not changed since 2017, I choosed to remove all source code related to logs sharing.

Please consider this as reserved for future usages.

verify_ssl_certificate

Type: boolean

Default value: False

Description: Activates or disables the verification of the SSL/TLS certificate when testing for URL.

Warning

If you set this index to True, you may get false positive result.

Indeed if the certificate is not registered to the CA or is simply invalid and the domain is still alive, you will always get INACTIVE as output.

debug

Type: dict

Description: Configures the debug / logging.

debug[active]

Type: boolean

Default value: False

Description: Activate or deactivates the debug / logging.

debug[level]

Type: string

Default value: info

Available values: INFO, DEBUG, WARNING, ERROR, CRITICAL.

Description: Activate or deactivates the debug / logging.

cli_decoding

Type: dict

Description: Configures everything related to the decoding of the given input sources.

cli_decoding[adblock]

Type: boolean

Default value: False

Description: Activates or disables the adblock decoding.

Note

If this index is set to True, every time we read a given file, we try to extract the elements that are present.

We basically only decode the adblock format.

Note

If this index is set to False, every time we read a given file, we will consider one line as an element to test.

cli_decoding[adblock_aggressive]

Type: boolean

Default value: False

Description: Activates or disables disable some aggressive settings related to the adblock decoding.

Warning

This option is available but please keep in mind that the underlying source code is still experimental.

cli_decoding[rpz]

Type: boolean

Default value: False

Description: Activates or disables the decoding of RPZ policies from each given input files.

cli_decoding[wildcard]

Type: boolean

Default value: False

Description: Activates or disables the decoding of wildcards for each given input files.

cli_testing

Type: dict

Description: Configures everything related to the CLI testing.

cli_testing[hosts_ip]

Type: string

Default value: "0.0.0.0"

Description: Sets the IP to prefix each lines of the hosts file.

cli_testing[max_workers]

Type: integer

Default value: null

Description: Sets the number of maximal processes workers that we are allowed to allocate for the testing.

Warning

If set to null, we use the default value calculated from your machine ressources. Meaning:

CPU cores - 2

cli_testing[autocontinue]

Type: boolean

Default value: True

Description: Activates or disables the automatic continuation subsystem.

cli_testing[inactive_db]

Type: boolean

Default value: True

Description: Activates or disables the usage of a “database” to store all INVALID and INACTIVE subject for continuous retest.

cli_testing[whois_db]

Type: boolean

Default value: True

Description: Activates or disables the uage of a “database” to store the expiration date of all domains with a valid expiration date.

Warning

We do not recomend you to disable this. In fact, this is your safety against the rate limite imposed by most WHOIS servers.

cli_testing[cidr_expand]

Type: boolean

Default value: False

Description: Activates or disables the expansion of CIDR formatted addresses.

cli_testing[complements]

Type: boolean

Default value: False

Description: Activate or disables the generation and test of the complements of a given subject.

Note

A complement is for example example.org if www.example.org is given and vice-versa.

cli_testing[cooldown_time]

Type:: float

Default value: 0.0

Description: Sets the cooldown time to apply between each test.

cli_testing[db_type]

Type: string

Default value: csv

Available values: csv, mariadb, mysql.

Description: Sets the database type (or engine) to use everytime we create a database or a storage of a potentially huge dataset.

cli_testing[file_filter]

Type: string

Default value: null

Description: A regular expression which we use to filter the subjects to (actually) test.

cli_testing[mining]

Type: boolean

Default value: True

Description: Activates or disables the mining subsystem.

cli_testing[local_network]

Type: boolean

Default value: False

Description: Activates or disables the consideration of the test(s) in or for a local or private network context.

cli_testing[preload_file]

Type: boolean

Default value: False

Description: Activates or disables the preloading of the given input files. When this is activates, we preload the given files into the auto continue subsystem dataset in order to optimize some of our processes regarding the auto continue.

Note

This option does not have any effect if the auto continue subsystem is disabled.

cli_testing[ci]

Type: dict

Description: Configures everything related to the Continuous Integration.

cli_testing[ci][active]

Type: boolean

Default value: False

Description: Activates or disables the Continuous Integration mechanism.

cli_testing[ci][commit_message]

Type: string

Default value: "PyFunceble - AutoSave"

Description: Sets the commit message to apply everytime we have to apply a commit except for the really last one.

cli_testing[ci][end_commit_message]

Type: string

Default value: "PyFunceble - Results"

Description: Sets the commit message to apply at the really end.

cli_testing[ci][max_exec_minutes]

Type: integer

Default value: 15

Description: Sets the number of minutes to wait before starting to stop a CI session.

Note

As many services are setting a rate limit per IP, it’s a good idea to set this value between 1 and 15 minute(s).

cli_testing[ci][branch]

Type: string

Default value: master

Description: Sets our git working branch. This is the branch from where we are supposed to store the tests (excepts the final results).

cli_testing[ci][distribution_branch]

Type: string

Default value: master

Description: Sets our git distributions branch. This is the branch from where we are supposed to store and push the final results.

cli_testing[ci][command]

Type: string

Default value: null

Description: Sets the command to execute before each commit (except the final one).

cli_testing[ci][end_command]

Type: string

Default value: null

Description: Sets the command to execute before the final commit.

cli_testing[display_mode]

Type: dict

Description: Configures everything related to what is displayed.

cli_testing[display_mode][dots]

Type: boolean

Default value: False

Description: Activate or disables the printing of dots or other characters when we skip the test of a subjec.

cli_testing[display_mode][dots]

Type: boolean

Default value: False

Description: Activate or disables the display of dots or other characters when we skip the test of a subjec.

cli_testing[display_mode][execution_time]

Type: boolean

Default value: False

Description: Activates or disables the display of the execution time.

cli_testing[display_mode][percentage]

Type: boolean

Default value: True

Description: Activates or disables the display and generation of the percentage - file - of each status.

cli_testing[display_mode][quiet]

Type: boolean

Default value: False

Description: Activates or disables the display of output to the terminal.

Warning

If the the dots mode is activate, this option will still allow them to work.

cli_testing[display_mode][less]

Type: boolean

Default value: True

Description: Activates or disables the display of the minimal information in the table we print to stdout.

cli_testing[display_mode][all]

Type: boolean

Default value: True

Description: Activates or disables the disply of the all information in the table we print to stdout.

cli_testing[display_mode][simple]

Type: boolean

Default value: False

Description: Activates or disables the simple output mode.

Note

When this mode is active, the system will only return the result in the following format: example.org ACTIVE.

cli_testing[display_mode][status]

Type: string | list

Default value: all

Available values: all, ACTIVE, INACTIVE, INVALID, VALID, SANE, MALICIOUS

Description: Sets the status that we are allowed to print to STDOUT.

Note

A list of status can be given if you want to filter multiple status at once.

cli_testing[display_mode][colour]

Type: boolean

Default value: True

Description: Activates or disables the coloration to STDOUT.

cli_testing[testing_mode]

Type: dict

Description: Configures the testing mode to apply.

Warning

Only one of those is take in consideration.

Here is the priority / checking order:

1. syntax
2. reputation
3. availability

cli_testing[testing_mode][availability]

Type: boolean

Default value: True

Description: Activates or disables the availability checker.

Note

This is the default mode.

cli_testing[testing_mode][syntax]

Type: boolean

Default value: True

Description: Activates or disables the syntax checker.

cli_testing[testing_mode][reputation]

Type: boolean

Default value: True

Description: Activates or disables the reputation checker.

cli_testing[days_between]

Type: dict

Description: Configures some days related events.

cli_testing[days_between][db_clean]

Type: integer

Default value: 28

Description: Sets the numbers of days since the introduction of a subject into the inactive dataset before it gets deleted.

Warning

As of PyFunceble 4.0.0 this is not actively implemented.

cli_testing[days_between][db_retest]

Type: integer

Default value: 28

Description: Sets the numbers of days since the introduction of a subject into the inactive dataset before it gets retested.

cli_testing[sorting_mode]

Type: dict

Description: Configures the sorting mode to apply.

Warning

Only one of those is take in consideration.

Here is the priority / checking order:

1. hierarchical
2. standard

cli_testing[sorting_mode][hierarchical]

Type: boolean

Default value: False

Description: Activates or disables the sorting of the files content (output) in a hierarchical order.

cli_testing[sorting_mode][standard]

Type: boolean

Default value: False

Description: Activates or disables the sorting of the files content (output) in our standard order.

cli_testing[file_generation]

Type: dict

Description: Configures everything related to the file generation.

cli_testing[file_generation][no_file]

Type: boolean

Default value: False

Description: Activates or disables the generation of any non-logs file(s).

cli_testing[file_generation][no_file]

Type: boolean

Default value: False

Description: Activates or disables the generation of any non-logs file(s).

cli_testing[file_generation][hosts]

Type: boolean

Default value: True

Description: Activates or disables the generation of the hosts file(s).

cli_testing[file_generation][plain]

Type: boolean

Default value: True

Description: Activates or disables the generation of the RAW file(s). What is meant is a list with only a list of subject (one per line).

cli_testing[file_generation][analytic]

Type: boolean

Default value: True

Description: Activates or disables the generation of the analytic file(s).

cli_testing[file_generation][unified_results]

Type: boolean

Default value: False

Description: Activates or disables the generation of the unified results file instead of the splitted one.

cli_testing[file_generation][merge_output_dirs]

Type: boolean

Default value: False

Description: Activates or disables the merging of the outputs of all inputted files inside a single subdirectory as opposed to the normal behavior.

lookup

Type: dict

Description: Configures everything related to the lookup tools to use while testing a given subject.

lookup[dns]

Type: boolean

Default value: True

Description: Activates or disables the usage of the DNS lookup whether possible.

lookup[http_status_code]

Type: boolean

Default value: True

Description: Activates or disables the usage of the HTTP status code whether possible.

lookup[netinfo]

Type: boolean

Default value: True

Description: Activates or disables the usage of the network information (or network socket) whether possible.

lookup[special]

Type: boolean

Default value: True

Description: Activates or disables the usage of our SPECIAL and extra rules whether possible.

lookup[whois]

Type: boolean

Default value: True

Description: Activates or disables the usage of the WHOIS record (or better said the expiration date in it) whether possible.

lookup[reputation]

Type: boolean

Default value: True

Description: Activates or disables the usage of the reputation dataset whether possible.

lookup[timeout]

Type: integer

Default value: 5

Description: Sets the default timeout to apply to each lookup utilities everytime it is possible to define a timeout.

dns

Type: dict

Description: Configures everything related to the DNS lookup.

dns[server]

Type: list

Default value: null

Description: Sets the DNS server(s) to work with.

Note

When a list is given the following format is expected.

dns_server:
  - dns1.example.org
  - dns2.example.org

Note

You can specify a port number to use to the DNS server if needed.

As example:

- 127.0.1.53:5353

Warning

Please be careful when you overwrite this option. If one is not correct, you can almost sure that all results are going to be flagged as INACTIVE.

dns[protocol]

Type: string

Default value: UDP

Available values: UDP, TCP, HTTPS, TLS.

Description: Sets the protocol to use for all DNS queries.

dns[follow_server_order]

Type: boolean

Default value: True

Description: Activates or disables the follow-up of the given order.

dns[trust_server]

Type: boolean

Default value: False

Description: Activates or disables the trust mode. When the trust mode is active and the first read DNS server gives us a negative response (without any error), we take it as it is.

Otherwise, when the trust mode is disabled, when the first read DNS server gives us a negative response (without any error), we still ask all other DNS servers that were given or found.

dns[delay]

Type: float

Default value: 0.0

Description: Sets the delay to apply between each DNS query.

user_agent

Type: dict

Description: Configures the user agent.

user_agent[browser]

Type: string

Default value: chrome

Available values: chrome, edge, firefox, ie, opera, safari.

Description: Sets the browser to get the get the latest user agent from.

Warning

This option is not taken in consideration if user_agent[custom] is not set to null.

user_agent[platform]

Type: string

Default value: linux

Available values: linux, macosx, win10

Description: Sets the platform to get the get the latest user agent for.

Warning

This option is not taken in consideration if user_agent[custom] is not set to null.

user_agent[custom]

Type: string

Default value: null

Description: Sets the user agent to use.

Warning

Setting this index will overwrite the choices made into user_agent[platform] and user_agent[browser].

http_codes

Type: dict

Description: Configures everything related to the HTTP status code and the way PyFunceble handles them.

http_codes[self_managed]

Type: bool

Default value: False

Description: Informs PyFunceble that the status code list should not be managed automatically.

http_codes[list]

Type: dict

Description: Categorizes the HTTP status codes.

http_codes[list][up]

Type: list

Default value:

- 100
- 101
- 102
- 200
- 201
- 202
- 203
- 204
- 205
- 206
- 207
- 208
- 226

Description: List the HTTP status codes which are considered as ACTIVE.

http_codes[list][potentially_down]

Type: list

Default value:

- 400
- 402
- 404
- 409
- 410
- 412
- 414
- 415
- 416
- 451

Description: List the HTTP status code which are considered as INACTIVE or POTENTIALLY_INACTIVE.

http_codes[list][potentially_up]

Type: list

Default value:

- 000
- 300
- 301
- 302
- 303
- 304
- 305
- 307
- 308
- 403
- 405
- 406
- 407
- 408
- 411
- 413
- 417
- 418
- 421
- 422
- 423
- 424
- 426
- 428
- 429
- 431
- 500
- 501
- 502
- 503
- 504
- 505
- 506
- 507
- 508
- 510
- 511

Description: List the HTTP status code which are considered as ACTIVE or POTENTIALLY_ACTIVE.

links

Type: dict

Description: Sets the list of links which can be used/called by the system when needed.

links[api_date_format]

Type: string

Default value: https://pyfunceble.funilrys.com/api/date-format

Description: Sets the link to use when we share logs.

links[api_no_referrer]

Type: string

Default value: https://pyfunceble.funilrys.com/api/no-referrer

Description: Sets the link to use when we share logs.