Indexes

This page will try to detail each configuration available into .PyFunceble.yaml along with the location of where we are looking for the configuration file.

adblock

Type: boolean

Default value: False

Description: Enable / disable the adblock format decoding.

Note

If this index is set to True, every time we read a given file, we try to extract the elements that are present.

We basically only decode the adblock format.

Note

If this index is set to False, every time we read a given file, we will consider one line as an element to test.

auto_continue

Type: boolean

Default value: True

Description: Enable / disable the auto continue system.

command

Type: string

Default value: ""

Description: Set the command to run before each commit (except the final one).

Note

The parsed command is called only if auto_continue and travis are set to True.

command_before_end

Type: string

Default value: ""

Description: Set the command to run before the final commit.

Note

The parsed command is called only if auto_continue and travis are set to True.

Note

Understand by final commit the commit which will deliver the last element we have to test.

custom_ip

Type: string

Default value: "0.0.0.0"

Description: Set the custom IP to use when we generate a line in the hosts file format.

Note

This index has no effect if generate_hosts is set to False.

days_between_db_retest

Type: integer

Default value: 1

Description: Set the number of day(s) between each retest of the INACTIVE and INVALID elements which are present into inactive_db.json.

Note

This index has no effect if inactive_database is set to False.

db_type

Type: string

Default value: json

Available values: json, mariadb, mysql, sqlite

Description: Set the database type to use everytime we create a database.

Note

This feature is applied to the following subsystems:

  • Autocontinue physically located (JSON) at output/continue.json.
  • InactiveDB physically located (JSON) at [config_dir]/inactive_db.json.
  • Mining physically located (JSON) at [config_dir]/mining.json.
  • WhoisDB physically located (JSON) at [config_dir]/whois.json.

debug

Type: boolean

Default value: False

Description: Enable / disable the generation of debug file(s).

Note

This index has no effect if logs is set to False

Warning

Do not touch this index unless you a have good reason to.

Warning

Do not touch this index unless you have been invited to.

dns_server

Type: None or list

Default value: null

Description: Set the DNS server(s) to work with.

Note

When a list is given the following format is expected.

dns_server:
  - dns1.example.org
  - dns2.example.org

Warning

We expect DNS server(s). If a non-DNS server is given. You’ll get almost all results as INACTIVE.

filter

Type: string

Default value: ""

Description: Set the element to filter.

Note

This index should be initiated with a regular expression.

generate_complements

Type: boolean

Default value: False

Description: Enable / disable the generation and test of the complements.

Note

A complement is for example example.org if www.example.org is given and vice-versa.

generate_hosts

Type: boolean

Default value: True

Description: Enable / disable the generation of the hosts file(s).

generate_json

Type: boolean

Default value: False

Description: Enable / disable the generation of the JSON file(s).

header_printed

Type: boolean

Default value: False

Description: Say to the system if the header has been already printed or not.

Warning

Do not touch this index unless you have a good reason to.

hierarchical_sorting

Type: boolean

Default value: False

Description: Say to the system if we have to sort the list and the outputs in a hierarchical order.

iana_whois_server

Type: string

Default value: whois.iana.org

Description: Set the server to call to get the whois referer of a given element.

Note

This index is only used when generating the iana-domains-db.json file.

Warning

Do not touch this index unless you a have good reason to.

idna_conversion

Type: boolean

Default value: False

Description: Tell the system to convert all domains to IDNA before testing.

Note

We use domain2idna for the conversion.

inactive_database

Type: boolean

Default value: True

Description: Enable / Disable the usage of a database to store the INACTIVE and INVALID element to retest overtime.

less

Type: boolean

Default value: True

Description: Enable / Disable the output of every information of screen.

local

Type: boolean

Default value: False

Description: Enable / Disable the execution of the test(s) in a local or private network.

logs

Type: boolean

Default value: True

Description: Enable / Disable the output of all logs.

maximal_processes

Type: integer

Default value: 25

Description: Set the number of maximal simultaneous processes to use/create/run.

mining

Type: boolean

Default value: True

Description: Enable / Disable the mining subsystem.

multiprocess

Type: boolean

Default value: False

Description: Enable / Disable the usage of multiple processes instead of the default single process.

no_files

Type: boolean

Default value: False

Description: Enable / Disable the generation of any file(s).

no_special

Type: boolean

Default value: False

Description: Enable / Disable the usage of the SPECIAL rules - which are discribes in the source column section.

no_whois

Type: boolean

Default value: False

Description: Enable / Disable the usage of whois in the tests.

plain_list_domain

Type: boolean

Default value: False

Description: Enable / Disable the generation of the plain list of elements sorted by statuses.

Warning

Do not touch this index unless you a have good reason to.

quiet

Type: boolean

Default value: False

Description: Enable / Disable the generation of output on the screen.

referer

Type: string

Default value: ""

Description: Set the referer of the element that is currently under test.

Warning

Do not touch this index unless you a have good reason to.

seconds_before_http_timeout

Type: integer

Default value: 3

Description: Set the timeout to apply to every HTTP status code request.

Note

This index must be a multiple of 3.

share_logs

Type: boolean

Default value: True

Description: Enable / disable the logs sharing.

Note

This index has no effect if logs is set to False.

show_execution_time

Type: boolean

Default value: False

Description: Enable / disable the output of the execution time.

show_percentage

Type: boolean

Default value: True

Description: Enable / disable the output of the percentage of each status.

simple

Type: boolean

Default value: False

Description: Enable / disable the simple output mode.

Note

If this index is set to True, the system will only return the result inf format: tested.element STATUS.

split

Type: boolean

Default value: True

Description: Enable / disable the split of the results files.

Note

Understand with “results files” the mirror of what is shown on screen.

syntax

Type: boolean

Default value: False

Description: Enable / disable the syntax (only) testing.

Warning

If this index is set to True, we ONLY check for syntax, not availability.

travis

Type: boolean

Default value: False

Description: Enable / disable the Travis CI autosaving system.

Warning

Do not activate this index unless you are using PyFunceble under Travis CI.

travis_autosave_commit

Type: string

Default value: "PyFunceble - AutoSave"

Description: Set the default commit message we want to use when have to commit (save) but our tests are not yet completed.

travis_autosave_final_commit

Type: string

Default value: "PyFunceble - Results"

Description: Set the default final commit message we want to use when we all tests are finished.

travis_autosave_minutes

Type: integer

Default value: 15

Description: Set the minimum of minutes we have to run before to automatically save our test results.

Note

As many services are setting a rate limit per IP, it’s a good idea to set this value between 1 and 15 minutes.

travis_branch

Type: string

Default value: master

Description: Set the git branch where we are going to push our results.

unified

Type: boolean

Default value: False

Description: Enable / Disable the generation of the unified results.

Note

This index has no effect if split is set to True.

user_agent

Type: string

Default value: "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"

Description: Set the User-Agent to use every time we are requesting something from a web server other than our API.

verify_ssl_certificate

Type: boolean

Default value: False

Description: Enable / Disable the verification of the SSL/TLS certificate when testing for URL.

Warning

If you set this index to True, you may get false positive result.

Indeed if the certificate is not registered to the CA or is simply invalid and the domain is still alive, you will always get INACTIVE as output.

whois_database

Type: boolean

Default value: True

Description: Enable / Disable the usage of the whois database to avoid/bypass whois server requests rate limit.

outputs

Type: dict

Description: Set the needed output tree/names.

Warning

If you choose to change anything please consider deleting our output/ directory and the dir_structure*.json files.

outputs[default_files]

Type: dict

Description: Set the default name of some important files.

outputs[default_files][dir_structure]

Type: string

Default value: dir_structure.json

Description: Set the default filename of the file which has the structure to re-construct.

Note

This index has no influence with dir_structure_production.json

outputs[default_files][iana]

Type: string

Default value: iana-domains-db.json

Description: Set the default filename of the file which has the formatted copy of the IANA root zone database.

outputs[default_files][inactive_db]

Type: string

Default value: inactive_db.json

Description: Set the default filename of the file which will save the list of elements to retest overtime.

outputs[default_files][results]

Type: string

Default value: results.txt

Description: Set the default filename of the file which will save the formatted copy of the public suffix database.

outputs[default_files][public_suffix]

Type: string

Default value: public-suffix.json

Description: Set the default filename of the file which will save the mirror of what is shown on screen.

outputs[default_files][mining]

Type: string

Default value: mining.json

Description: Set the default filename of the file which will save the temporary list of mined subject to test.

outputs[default_files][whois_db

Type: string

Default value: whois_db.json

Description: Set the default filename of the file which will save the whois information for caching.

outputs[default_files][sqlite]

Type: string

Default value: pyfunceble.db

Description: Set the default filename of the file which will save all database in sqlite format.

outputs[db_type]

Type: dict

Description: Set the default name of some important files related to the db_type index.

outputs[db_type][directory]

Type: string

Default value: .db_type/

Description: Set the default directory where we have to save the SQL files which we are later going to be used for database generation/creation.

outputs[db_type][files]

Type: dict

Description: Set the default name of some important files related to the db_type index.

outputs[db_type][files][sqlite]

Type: string

Default value: sqlite.sql

Description: Set the default name of the file to call when we have to construct the SQLite database.

outputs[domains]

Type: dict

Description: Set the default name of some important files related to the plain_list_domain index.

outputs[domains][directory]

Type: string

Default value: domains/

Description: Set the default directory where we have to save the plain list of elements for each status.

outputs[domains][filename]

Type: string

Default value: list

Description: Set the default filename of the file which will save the plain list of elements.

outputs[hosts]

Type: dict

Description: Set the default name of some important files related to the generate_hosts index.

outputs[hosts][directory]

Type: string

Default value: hosts/

Description: Set the default directory where we have to save the hosts files of the elements for each status.

outputs[hosts][filename]

Type: string

Default value: hosts

Description: Set the default filename of the file which will save the hosts files of the elements.

outputs[json]

Type: dict

Description: Set the default name of some important files related to the generate_json index.

outputs[json][directory]

Type: string

Default value: json

Description: Set the default directory where we have to save the JSON files of the elements for each status.

outputs[json][filename]

Type: string

Default value: dump.json

Description: Set the default filename of the file which will save the JSON files of the elements.

outputs[complements]

Type: dict

Description: Set the default name of some important files/directories related to the generate_complements index.

outputs[complements][directory]

Type: string

Default value: complements

Description: Set the default directory where we have to save the complements related files sorted by status.

outputs[analytic]

Type: dict

Description: Set the default name of some important files and directories related to the generate_hosts index.

outputs[analytic][directories]

Type: dict

Description: Set the default name of some important directories related to the http_codes[active] index.

outputs[analytic][directories][parent]

Type: string

Default value: Analytic/

Description: Set the default directory where we are going to put everything related to the HTTP analytic.

outputs[analytic][directories][potentially_down]

Type: string

Default value: POTENTIALLY_INACTIVE/

Description: Set the default directory where we are going to put all potentially inactive data.

outputs[analytic][directories][potentially_up]

Type: string

Default value: POTENTIALLY_INACTIVE/

Description: Set the default directory where we are going to put all potentially active data.

outputs[analytic][directories][up]

Type: string

Default value: POTENTIALLY_INACTIVE/

Description: Set the default directory where we are going to put all active data.

outputs[analytic][directories][suspicious]

Type: string

Default value: SUSPICIOUS/

Description: Set the default directory where we are going to put all suspicious data.

outputs[analytic][filenames]

Type: dict

Description: Set the default name of some important files related to the http_codes[active] index and the HTTP analytic subsystem.

outputs[analytic][filenames][potentially_down]

Type: string

Default value: down_or_potentially_down

Description: Set the default filename where we are going to put all potentially inactive data.

outputs[analytic][filenames][potentially_up]

Type: string

Default value: potentially_up

Description: Set the default filename where we are going to put all potentially active data.

outputs[analytic][filenames][up]

Type: string

Default value: active_and_merged_in_results

Description: Set the default filename where we are going to put all active data.

outputs[analytic][filenames][suspicious]

Type: string

Default value: suspicious_and_merged_in_results

Description: Set the default filename where we are going to put all suspicious data.

outputs[logs]

Type: dict

Description: Set the default name of some important files and directories related to the logs index.

outputs[logs][directories]

Type: dict

Description: Set the default name of some important directories related to the logs index.

outputs[logs][directories][date_format]

Type: string

Default value: date_format/

Description: Set the default directory where we are going to put everything related to the data when the dates are in the wrong format.

outputs[logs][directories][no_referer]

Type: string

Default value: no_referer/

Description: Set the default directory where we are going to put everything related to the data when no referer is found.

outputs[logs][directories][parent]

Type: string

Default value: no_referer/

Description: Set the default directory where we are going to put everything related to the data when no referer is found.

outputs[logs][directories][percentage]

Type: string

Default value: percentage/

Description: Set the default directory where we are going to put everything related to percentages.

outputs[logs][directories][whois]

Type: string

Default value: whois/

Description: Set the default directory where we are going to put everything related to whois data.

Note

This is the location of all files when the debug index is set to True.

outputs[logs][filenames]

Type: dict

Description: Set the default filenames of some important files related to the logs index.

outputs[logs][filenames][auto_continue]

Type: string

Default value: continue.json

Description: Set the default filename where we are going to put the data related to the auto continue subsystem.

Note

This file is allocated if the auto_continue is set to True.

outputs[logs][filenames][execution_time]

Type: string

Default value: execution.log

Description: Set the default filename where we are going to put the data related to the execution time.

Note

This file is allocated if the show_execution_time is set to True.

outputs[logs][filenames][percentage]

Type: string

Default value: percentage.txt

Description: Set the default filename where we are going to put the data related to the percentage.

Note

This file is allocated if the show_percentage is set to True.

outputs[main]

Type: string

Default value: ""

Description: Set the default location where we have to generate the parent_directory directory and its dependencies.

outputs[parent_directory]

Type: string

Default value: output/

Description: Set the directory name of the parent directory which will contain all previously nouned directories.

outputs[splited]

Type: dict

Description: Set the default name of some important files and directory related to the split index.

outputs[splited][directory]

Type: string

Default value: splited/

Description: Set the default directory name where we are going to put the split data.

status

Type: dict

Description: Set the needed, accepted and status name.

status[list]

Type: dict

Description: Set the needed and accepted status name.

Warning

All status should be in lowercase.

status[list][valid]

Type: list

Default value: ["valid","syntax_valid","valid_syntax"]

Description: Set the accepted VALID status.

Note

This status is only shown if the syntax index is activated.

status[list][up]

Type: list

Default value: ["up","active"]

Description: Set the accepted ACTIVE status.

status[list][generic]

Type: list

Default value: ["generic"]

Description: Set the accepted generic status.

Note

This status is the one used to say the system that we have to print the complete information on the screen.

status[list][http_active]

Type: list

Default value: ["http_active"]

Description: Set the accepted status for the outputs[analytic][filenames][up] index.

status[list][down]

Type: list

Default value: ["down","inactive", "error"]

Description: Set the accepted status INACTIVE index.

status[list][invalid]

Type: list

Default value: ["ouch","invalid"]

Description: Set the accepted status INVALID index.

status[list][potentially_down]

Type: list

Default value: ["potentially_down", "potentially_inactive"]

Description: Set the accepted status for the outputs[analytic][filenames][potentially_down] index.

status[list][potentially_up]

Type: list

Default value: ["potentially_up", "potentially_active"]

Description: Set the accepted status for the outputs[analytic][filenames][potentially_up] index.

status[list][suspicious]

Type: list

Default value: ["strange", "hum", "suspicious"]

Description: Set the accepted status for the outputs[analytic][filenames][suspicious] index.

status[official]

Type: dict

Description: Set the official status name.

Note

Those status are the ones that are printed on the screen.

Warning

After any changes here please delete dir_structure.json and the output/ directory.

status[official][up]

Type: string

Default value: ACTIVE

Description: Set the returned status for the ACTIVE case.

status[official][down]

Type: string

Default value: INACTIVE

Description: Set the returned status for the INACTIVE case.

status[official][invalid]

Type: string

Default value: INVALID

Description: Set the returned status for the INVALID case.

status[official][valid]

Type: string

Default value: VALID

Description: Set the returned status for the VALID case.

Note

This status is only shown if the syntax index is activated.

http_codes

Type: dict

Description: Handle the interpretation of each status codes when we do and generate our analytic data.

http_codes[active]

Type: boolean

Default value: True

Description: Enable / Disable the usage of the HTTP status code extraction.

http_codes[list]

Type: dict

Description: Categorize the http status code as mentioned in the documentation related to the HTTP Code column.

http_codes[list][up]

Type: list

Default value:
- 100
- 101
- 200
- 201
- 202
- 203
- 204
- 205
- 206

Description: List the HTTP status codes which are considered as ACTIVE.

http_codes[list][potentially_down]

Type: list

Default value:
- 400
- 402
- 403
- 404
- 409
- 410
- 412
- 414
- 415
- 416

Description: List the HTTP status code which are considered as INACTIVE or POTENTIALLY_INACTIVE.

http_codes[list][potentially_up]

Type: list

Default value:
- 000
- 300
- 301
- 302
- 303
- 304
- 305
- 307
- 403
- 405
- 406
- 407
- 408
- 411
- 413
- 417
- 500
- 501
- 502
- 503
- 504
- 505

Description: List the HTTP status code which are considered as ACTIVE or POTENTIALLY_ACTIVE.