Usage

PyFunceble Python API

If you are working with a python script, module or even class, you can integrate PyFunceble to your main logic by importing it and using its API (cf: API Documentation).

This section will present some example of the way you can interact with PyFunceble from anything written in Python.

Check the availability of a domain

from PyFunceble import DomainAvailabilityChecker


checker = DomainAvailabilityChecker()
to_test = "github.com"

# You can do it this way.
status = checker.set_subject(to_test).get_status()

# Or this way.
checker.set_subject(to_test)
status = checker.get_status()

# We can convert the status to json.
status_json = status.to_json()

# We can convert the status to dict.
status_dict = status.to_dict()

# We can ask "questions".
print(f"Is {to_test} ACTIVE ?", status.is_active())
print(f"Is {to_test} INACTIVE ?", status.is_inactive())
print(f"Is {to_test} INVALID ?", status.is_invalid())

Check the availability of an IP

from PyFunceble import IPAvailabilityChecker


checker = IPAvailabilityChecker()

to_test = "192.0.2.1"

# You can do it this way.
status = checker.set_subject(to_test).get_status()

# Or this way.
checker.set_subject(to_test)
status = checker.get_status()

# We can convert the status to json.
status_json = status.to_json()

# We can convert the status to dict.
status_dict = status.to_dict()

# We can ask "questions".
print(f"Is {to_test} ACTIVE ?", status.is_active())
print(f"Is {to_test} INACTIVE ?", status.is_inactive())
print(f"Is {to_test} INVALID ?", status.is_invalid())

Check the availability of an IP or domain

from PyFunceble import DomainAndIPAvailabilityChecker


checker = DomainAndIPAvailabilityChecker()

to_test = "github.com"

# You can do it this way.
status = checker.set_subject(to_test).get_status()

# Or this way.
checker.set_subject(to_test)
status = checker.get_status()

# We can convert the status to json.
status_json = status.to_json()

# We can convert the status to dict.
status_dict = status.to_dict()

# We can ask "questions".
print(f"Is {to_test} ACTIVE ?", status.is_active())
print(f"Is {to_test} INACTIVE ?", status.is_inactive())
print(f"Is {to_test} INVALID ?", status.is_invalid())

Check the availability of URL

from PyFunceble import URLAvailabilityChecker


checker = URLAvailabilityChecker()

to_test = "http://github.com/PyFunceble"

# You can do it this way.
status = checker.set_subject(to_test).get_status()

# Or this way.
checker.set_subject(to_test)
status = checker.get_status()

# We can convert the status to json.
status_json = status.to_json()

# We can convert the status to dict.
status_dict = status.to_dict()

# We can ask "questions".
print(f"Is {to_test} ACTIVE ?", status.is_active())
print(f"Is {to_test} INACTIVE ?", status.is_inactive())
print(f"Is {to_test} INVALID ?", status.is_invalid())

Check the syntax of domains

from PyFunceble import DomainSyntaxChecker


checker = DomainSyntaxChecker()

to_test = "github.com"

# You can do it this way.
status = checker.set_subject(to_test).get_status()

# Or this way.
checker.set_subject(to_test)
status = checker.get_status()

# We can convert the status to json.
status_json = status.to_json()

# We can convert the status to dict.
status_dict = status.to_dict()

# We can ask "questions".
print(f"Is {to_test} VALID ?", status.is_valid())
print(f"Is {to_test} INVALID ?", status.is_invalid())

Check the syntax of IP (v4 or v6)

from PyFunceble import IPSyntaxChecker


checker = IPSyntaxChecker()

to_test = "192.0.2.1"

# You can do it this way.
status = checker.set_subject(to_test).get_status()

# Or this way.
checker.set_subject(to_test)
status = checker.get_status()

# We can convert the status to json.
status_json = status.to_json()

# We can convert the status to dict.
status_dict = status.to_dict()

# We can ask "questions".
print(f"Is {to_test} VALID ?", status.is_valid())
print(f"Is {to_test} INVALID ?", status.is_invalid())

Check the syntax of URLs

from PyFunceble import URLSyntaxChecker


checker = URLSyntaxChecker()

to_test = "https://github.com/PyFunceble"

# You can do it this way.
status = checker.set_subject(to_test).get_status()

# Or this way.
checker.set_subject(to_test)
status = checker.get_status()

# We can convert the status to json.
status_json = status.to_json()

# We can convert the status to dict.
status_dict = status.to_dict()

# We can ask "questions".
print(f"Is {to_test} VALID ?", status.is_valid())
print(f"Is {to_test} INVALID ?", status.is_invalid())

From a terminal

This chapter also relates to writing scripts in bash and PowerShell as the uses the same syntaxes.

--show-completion "shell"

Show shell completion script and exit.

Available Values: bash, zsh

Note

This argument provides the autocompletion script that you can use to get access to the autocompletion assistance.

It is meant to be used like this:

$ source <(pyfunceble --show-completion bash)
$ pyfunceble --do[TAB]
--domain  --dots

--help

Show the help message and exit.


-v | --version

Show the version of PyFunceble and exit.


Test sources

-d "$DOMAIN" | --domain "$DOMAIN"

This argument takes one or more values separated by spaces.

Test one or more :code:`$DOMAIN`s,

$ PyFunceble -d example.org example.net

A domain is defined as it do NOT start with a protocol:// and it do not contain a forward slash /

This is a domain: example.org

Note

When this option is used, no output files are generated.


-u "$URI" | --url "$URI"

Test one or more full URL, separated by spaces.

This argument takes one or more values.

$ PyFunceble -url https://example.org/AlIvE https://example.com/GoNe

Note

When we test the availability of a URL, we (only) check the HTTP status code of the given URL.

A URI or URL is defined by it is starting with a protocol.

This is a URL http://example.org/?example=yes

This is another URL https://example.org

This is another URL ftp://ftp.example.org


-f "$DOMAIN" | --file "$DOMAIN_FILE"

Read a local or remote (RAW link) file and test all domains inside it. If remote (RAW link) file is given, PyFunceble will download it, and test the content of the given RAW link as if it was a locally stored file.

$ PyFunceble -f "$DOMAIN"_1 "$DOMAIN"_2
$ PyFunceble -f "$DOMAIN_FILE"_1 "$DOMAIN_FILE"_2
$ PyFunceble --file "$DOMAIN_FILE"_1 "$DOMAIN_FILE"_2

Note

  • This argument takes one or more space separated values.
  • You can combine -f and -uf in the same test.

Warning

You can not combine the usage of -f, -uf with --adblock at the same time


-uf "$URL_FILES" | --url-file "$URL_FILES"

Read a local or remote (RAW link) file and test all (full) URLs inside it. If remote (RAW link) file is given, PyFunceble will download it, and test the content of the given RAW link as if it was a locally stored file.

This argument test if a URL which is inside the given file is available. It ONLY tests full URLs.

$ PyFunceble -uf "$URI"_1 "$URI"_2

When a remote located source is provided, we will download the given URL and test its content assuming that each line represents a URL to test.

$ PyFunceble -uf "$URL_FILES"

Note

  • This argument takes one or more space separated values.
  • You can combine -f and -uf in the same test.
  • We consider one line as one URL to test.

Warning

A test with this argument consists of the comparison of the status code. No WHOIS record will be requested nor DNS Lookup will be done.

You can not combine the usage of -f, -uf and --adblock at the same time


Source filtering, decoding, conversion and expansion

--adblock

This feature is used to extract all domains, IPv4 and IPv6 addresses from a adblock formatted file and test the status and validate the extracted domains.

To use this feature you’ll need to set the -f "$DOMAIN" | --file "$DOMAIN_FILE" to tell PyFunceble from where to obtain the given list.

$ pyfunceble --adblock -f "$ADBLOCK_FILES"

Default value: adblock: False

Warning

You can not combine the usage of -f, -uf and --adblock at the same time


--cidr

This feature will expand CIDR formatted addresses.

Default value: cidr_expand: False


--complements

A complement is for example example.org if www.example.org is given and vice-versa.

Default value: complements: False


--filter "RegEx"

A Regex string to match in order to test a given line.

Default value: file_filter: null

If you only want to test all blogspot URI or domains from your list, this argument allows you to do that!

$ pyfunceble --filter '^\.blogspot\.(com|net)$' -f $DOMAIN_FILE

--mining

Todo

Find out more about how this actually works…

Want to find domain or URL linked to a domain in your list? This argument will exactly do that.

Default value: mining: False


--rpz

New in version 3.3.3.

Activates or disables the decoding of RPZ policies from each given input source (-f).

$ pyfunceble --rpz -f $RPZ_FILES

The --rpz is used to test domains from a fully functional and valid RPZ (Response Policy Zone). If you do provide the required zone SOA record it will extract the right domains to test.

Example of a fully functional RPZ zone

spyware.my-rpz.internal.   86400   IN      SOA     my.awesome.rps.zone. need.to.know.only. 2021011401 300 60 604800 3600
*.360.com.spyware.my-rpz.internal. 86400   IN      CNAME   .
*.360safe.com.cn.spyware.my-rpz.internal.  86400   IN      CNAME   .
*.360totalsecurity.com.spyware.my-rpz.internal.    86400   IN      CNAME   .
360.com.spyware.mypdns.cloud.   86400   IN      CNAME   .
360safe.com.cn.spyware.mypdns.cloud.    86400   IN      CNAME   .
360totalsecurity.com.spyware.mypdns.cloud.      86400   IN      CNAME   .

(PS. RPZ zones does not requires the NS records RFC 1034)

From the example above PyFunceble will be testing the following domains.

You can make a simple test with the above zone example by copy/pasting.

In case your RPZ zone are missing the required SOA entry, you should consider combining the --rpz with –wildcard to avoid all your wildcard’s domain becoming marked as INVALID

Default value: rpz: False

Warning

You can currently not use the --rpz in combination with --syntax to validate or syntax test a rpz formatted file.

See also

See discussions 149 for more information and participate in it’s development.


--wildcard

New in version 3.3.0.

The flag to use when your source(-f) of domains starts with a wildcard.

This flag will subtract the *.$DOMAIN and test the $DOMAIN according to the test arguments given.

Default value: wildcard: False

As examples of when to use this argument. The first one will return INVALID if --wildcard is not set to true.

This feature is related to the –rpz

'*.example.org'
'example.org'

Test control

--cooldown-time

Sets a cooldown time (in second) to be applied between (sleep) before/between each test cycles is done.

Default value: cooldown_time: 0.0


--local

Activates or disables the consideration of the test(s) in or for a local or private network context.

Want to run a test over a local or private network? This argument will disable the limitation which does not apply to private networks.

Default value: local_network: False


--dns-lookup

Activates or disables the usage of the DNS lookup whether possible.

Default value: dns: True

Don’t want to perform some DNS lookup? This argument is for you.


--http-status-code-lookup | --http

Don’t want to take the result of the HTTP code execution into consideration?

This argument allows you to disable the HTTP status code checker!

Default value: http_status_code: True


--netinfo-lookup

Activates or disables the usage of the network information (or network socket) whether possible.

Don’t want to perform some netinfo lookup ? This argument is for you.

Default value: netinfo: True


--special-lookup

Activates or disables the usage of our SPECIAL and extra rules whether possible.

Don’t want to use/apply the Special Rules - which are explained in the source column section?

This argument disables them all.

Default value: special: True


--whois-lookup

New in version 4.0.0.

Activates or disables the usage of the WHOIS record (or better said the expiration date in it) when possible.

Don’t want to use or take the whois date into consideration? This argument allows you to disable it!

Default value: whois: True

Note

When you use the --syntax no WHOIS data lookup will be performed In other words: --syntax overrules this argument


--reputation-lookup

Todo

Check which of the reputation is alive or the code difference

Want to take the reputation data into consideration?

Activates or disables the usage of the reputation dataset when possible.

Default value: reputation: False


--reputation

Activates or disables the reputation checker.

Default value: False


--syntax

This code is to check the syntax of domains when the -f and URI’s when –url is used as source.

You should be able to use both -f and code:-uf` at the same time with --syntax

When you are using this flags there will not be performed any other test, such as the WHOIS or HTTP status code

Default value: syntax: False

Note

TIP: If you would like to gain some serious performance while testing with --syntax, We recommend you disable –auto-continue

See note for --rpz


-t "seconds" | --timeout "seconds"

Sets the default timeout to apply to each lookup utilities every time it is possible to define a timeout.

Default value: timeout: 5 seconds


-ua "full string" | --user-agent "full string"

User defined user agent to use in the http status code lookup.

user_agent:
    browser: chrome
    platform: linux

Warning

If not given, we try to get the latest (automatically) for you

Example of how to change the default from CLI.

$ pyfunceble --user-agent "Mozilla/5.0 (X11; U; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36"

-vsc | --verify-ssl-certificate

Activates or disables the verification of the SSL/TLS certificate when testing for URL.

Default value: verify_ssl_certificate: False

Warning

If you activate the verification of the SSL/TLS certificate, you may get false-positive results.

Indeed if the certificate is not registered to the CA or is simply invalid and the domain is still alive, you will always get INACTIVE as output.


DNS control

--dns

By default, PyFunceble will use the system-wide DNS settings. This can be changed with the ability to configure which DNS-Servers you like PyFunceble to use during the test.

You set this up with the CLI command --dns or insert it into your personal .PyFunceble.yaml

You can add several separated by spaces and they will all be used in a order. (Kind of Round Robin style)

Default value: Follow OS DNS ==> server: null

$ pyfunceble -dns 127.0.1.53:5303 127.0.0.1 -f $DOMAIN_FILE

You can also set default DNS servers used for testing within the my_project/.PyFunceble.yaml file. (No secondary indent)

server:
- 1.2.3.4
- 5.6.7.8
- 9.10.11.12:5302

Warning

We expect a DNS server(s). If you add this flag but no DNS server(s) is given. You’ll almost for certain get all results as INACTIVE

This could happen in case you use --dns -f

Note

You can specify the port number to be used on the DNS server if needed.


--dns-protocol

Sets the protocol to use for the DNS queries.

Default value: protocol: UDP

Available values: UDP, TCP, HTTPS, TLS. Case-Sensitive

$ pyfunceble --dns doh.powerdns.org --dns-protocol HTTPS -f $DOMAIN_FILE
$ pyfunceble --dns 192.0.2.2:53 --dns doh.powerdns.org --dns-protocol HTTPS

--follow-server-order

New in version 4.0.0.

Let us follow or mix the order of usage of the given or found DNS server(s).

Default value: True


--trust-dns-server

New in version 4.0.0.

Activates or disable the trust mode.

Default value: False

Note

When active, when the first read DNS server give us a negative response - without error - we take it as it it.

Otherwise, if not active, when the first read DNS server give us a negative response - without error - we still consolidate by checking all given/found server.


--dns-delay

New in version 4.0.0.

Sets the delay to apply between each DNS query.

Default value: 0.0

Note

When greater that 0.0, a delay will be applied between each DNS query.

Otherwise, if equal to 0.0, no delay will be applied.


Databases

--inactive-database

Switch the value of the usage of a database to store inactive domains of the currently tested list.

Default value: db_clean: 28 Day(s).

This argument will disable or enable the usage of a database which saves all INACTIVE and INVALID domain of the given file over time.


--database-type

Sets the database engine to use.

Default value: db_type: csv

Available values: csv, mariadb, mysql.


--inactive-db

Activates or disables the usage of a ‘database’ to store all ‘INACTIVE’ and ‘INVALID’ subject for continuous retest.

Configured value: inactive_db: True


-dbr "time" | --days-between-db-retest "time"

Sets the numbers of days since the introduction of a subject into the inactive dataset before it gets retested.

Default value: db_retest: 1 Day(s)

Note

This argument is only used if -db or inactive_database : true (under .PyFunceble.yaml) are activated. See also –inactive-db


-wdb | --whois-database

Activates or disables the usage of a “database” to store the expiration date of all domains with a valid expiration date.

Default value: whois_db: True


Output control

-a | --all

Activates or disables the display of the all information in the table we print to stdout (screen).

Default value: all: False

Default:

Domain                        Status      Source
----------------------------- ----------- ----------
pyfunceble.readthedocs.io     ACTIVE      SYNTAX

When :code:`all: True`:

Domain                        Status      Expiration Date   Source     HTTP Code  Checker
----------------------------- ----------- ----------------- ---------- ---------- -------------
pyfunceble.readthedocs.io     ACTIVE      Unknown           NSLOOKUP   302        AVAILABILITY

--color | --colour

Activates or disables the coloration to STDOUT.

Default value: colour: True

Don’t want any colour ? This argument is for you!


--display-status

New in version 4.0.0.

Sets the status that we are allowed to print to stdout.

Multiple space separated statuses can be given.

Default value: status: all

Available values: all, ACTIVE, INACTIVE, INVALID, VALID, SANE, MALICIOUS

Default response

$ pyfunceble -d google-analytics.com duckduckgo.com --whois-lookup

Subject                                              Status      Source
---------------------------------------------------- ----------- ----------
duckduckgo.com                                       ACTIVE      DNSLOOKUP
google-analytics.com                                 INACTIVE    STDLOOKUP

Show only active and inactive

$ pyfunceble -d google-analytics.com duckduckgo.com --whois-lookup \
--display-status INACTIVE ACTIVE

Subject                                              Status      Source
---------------------------------------------------- ----------- ----------
duckduckgo.com                                       ACTIVE      DNSLOOKUP
google-analytics.com                                 INACTIVE    STDLOOKUP

Show only inactive

$ pyfunceble -d google-analytics.com duckduckgo.com --whois-lookup \
  --display-status INACTIVE

Subject                                              Status      Source
---------------------------------------------------- ----------- ----------
google-analytics.com                                 INACTIVE    STDLOOKUP

Note

If you have provided more than one $DOMAIN_FILE as input source, then the printed status will be in same order as your $DOMAIN_FILE was given in the input.

For an example you can visit: github


-ex | --execution

Want to know the execution time of your test? Well, this argument will let you know!

Default value: execution_time: False


--hierarchical

Activates or disables the sorting of the files content (output) in a hierarchical order.

Default value: hierarchical: False

This argument will output the result listed in a hierarchical order.


-h | --hosts

This argument will let the system know if it want to generate a hosts formatted result file for each status.

Default value: hosts: True

See also

–plain, –no-files

Note

There is an ongoing request to set the default value of hosts: False You should be following this issue as it might affect your setup/results later on. Flip defaults for host


-ip "ip-address" | --hosts-ip “ip-address”

Sets the IP to prefix each lines of the hosts file.

Default value: 0.0.0.0


--logging-level

New in version 4.0.0.

You can configure the logging level to be outputted in STDOUT (screen) when you uses --no-files. Default outputs to output/*_logging_/**.log

Optional values. (From less to more information)

  • --logging-level critical ==> CRITICAL
  • --logging-level error ==> ERROR
  • --logging-level info ==> INFO (default)
  • --logging-level warning ==> WARNING
  • --logging-level debug ==> DEBUG

--merge-output

Activates or disables the merging of the outputs of all inputted files inside a single subdirectory as opposed to the normal behavior.

Default value: merge_output_dirs: False


--no-files

Activates or disables the generation of any non-logs and status file(s).

Default value: no_file: False


--output-location

New in version 4.0.0.

This is used to direct the output location and matches PYFUNCEBLE_OUTPUT_LOCATION.

With this new option you no longer need to add the Global Variable but can append it directly to the CLI string.

$ pyfunceble --output-location /tmp/pyfunceble -f $DOMAIN_FILE

--unified-results

Activates or disables the generation of the unified results file instead of the divided output in individual subfolder under output/.

Default value: unified_results: False

This argument disables the generation of the result.txt file.


--percentage

Activates or disables the display and generation of the percentage - file - of each status.

Default value: percentage: True

This argument will disable or enable the generation of the percentage of each status.


--plain

Activates or disables the generation of the generation of clean file(s).

This will output a file per status only containing the subject(s). (One record per line)

Default value: plain: True


--dots

CLI only: Activate or disables the display of dots or other characters when we skip the test of a subject.

CI only: If you combine the --ci --dots we display a dot for each record we tests.

Default value: dots: False


-q | --quiet

Activates or disables the display of output to the terminal.

Default value: quiet: False


--share-logs

Want to help make PyFunceble a better tool?

Then you can share your logs with our backend API which collect all logs!

Default value: share_logs: False

Changed in version 4.0.0.


-s | --simple

Activates or disables the simple output mode.

Default value: simple: False

Want as less as possible data on screen? This argument returns as less as possible on screen!


Multiprocessing

-w | --max-workers

New in version 4.0.0.

Sets the number of maximal worker to use.

Keep in mind that the --max-workers mostly - if not only - affects the number of tester sub-processes. Because we want to safely write the files, we still need a single process which read the submitted results and generate the outputs.

The reason we added this to PyFunceble 4.0.0 is we don’t want to have a wrongly formatted output file.

If you have more than 2 CPU cores/processes the default will be number of CPU - 2. Otherwise, it will 1.

Default value: max_workers: null

Note

If you have a CPU with 4 cores or Threads (depends on it’s age) Then the number of workers will be 4 - 2 = 2 workers

Warning

This section about max-workers is still under construction, but it is close to how it is working.

  • This means you should be experimenting a bit your self.

To follow the “behind the scene” talk about the subject, please take a look at issue


CI / CD

--ci

Activates or disables the Continuous Integration mechanism.

Default value: active: False

Note

If you combine this argument with the --quiet argument, the test will output a dotted line, where each dot (.) represent one test result or input which was skipped because it was previously tested.

Want to use PyFunceble under a supported CI infrastructure/network? This argument is suited for your needs!


--ci-max-minutes

Sets the number of minutes to wait before starting to stop a CI session.

Default value: max_exec_minutes: 15


--ci-branch

Sets our git working branch. This is the branch from where we are supposed to store the tests (excepts the final results).

Default value: branch: master

Note

Currently the branch need to exist, but there are being worked on a path to have PyFunceble to create the sub-branch and finally merge it into the --ci-distribution-branch


--ci-distribution-branch

Sets our git distributions branch. This is the branch from where we are supposed to store and push the final results.

Default value: distribution_branch: master

Note

The difference between this and --ci-branch is the fact that this branch will get the (final) result only when the test is finished under the given --ci-branch.

As an example, this allows us to have 2 branches:

--ci-branch processing # (CI branch), for the tests with PyFunceble.
--ci-distribution-branch master # (CI distribution branch), for the
                                # distribution of the results of PyFunceble.

--ci-command "something"

Changed in version 4.0.0.

Sets the command to execute before each commit (except the final one).

Default value: command: null

Note

In this example, something should be a script or a program which have to be executed when we reached the end of the given file.

Note

This argument is only used if --ci or ci: true (under .PyFunceble.yaml) are activated.


--ci-end-command "something"

Changed in version 4.0.0.

Sets the command to execute before the final commit.

Default value: end_command: null

Note

In this example, something should be a script or a program which have to be executed when we reached the end of the given file.

Note

This argument is only used if --ci or ci: true (under .PyFunceble.yaml) are activated.


--ci-commit-message "message"

Changed in version 4.0.0.

Sets the commit message to apply every time we have to apply a commit except for the really last one.

Default value: commit_message: "PyFunceble - AutoSave"

This argument allows us to set a custom commit message which is going to be used as a commit message when saving.

Note

This argument is only used if --ci or ci: true (under .PyFunceble.yaml) are used.

Note

This argument is only used if we have to split the work into multiple processes because a list is too long or the timeout is reached.

Warning

Please avoid the usage of [ci skip] here.


--ci-end-commit-message

Changed in version 4.0.0.

Sets the commit message to apply at the really end.

Default value: end_commit_message: "PyFunceble - Results"

Note

This argument is only used if --ci or ci: true (under .PyFunceble.yaml) are used.

Note

This argument is only used if we reached the end of the list we are or have to test.


-c | --auto-continue | --continue

This argument is to used for auto-continuing from a previously under CI

Default value: autocontinue: False

This argument activates or deactivates the auto-continue subsystem. Indeed, as we can automatically continue if the script has been stopped, this switch allows us to disable or enable the usage of that specific subsystem.


--preload

New in version 4.0.0.

Activates or disables the preloading of the input file(s) into the continue dataset before starting the tests.

The –preload argument - or its option counterpart - ping is given, we decode and load the given input files into the continue dataset before starting the test.

This reduces the waiting time while continuing a previous session.

Note

This argument is useless unless the auto continue subsystem is active.

The preloading may take some time depending of the size of the file to test, but this is the price for a smooth and better autocontinue. Especially under CI’s.


Global Variables

Here is the list of environment variables we use and how we use them if they are set.

Note

If used in a script like bash or a terminal directly you have to use the export as PyFunceble is running as sub-processes

Environment Variable How to use them?
PYFUNCEBLE_AUTO_CONFIGURATION Tell us if we have to install/update the configuration file automatically.
PYFUNCEBLE_DB_CHARSET Tell us the MariaDB charset to use.
PYFUNCEBLE_DB_HOST Tell us the host or the Unix socket (absolute file path) of the MariaDB database.
PYFUNCEBLE_DB_NAME Tell us the name of the MariaDB database to use.
PYFUNCEBLE_DB_PASSWORD Tell us the MariaDB user password to use.
PYFUNCEBLE_DB_PORT Tell us the MariaDB connection port to use.
PYFUNCEBLE_DB_USERNAME Tell us the MariaDB user-name to use.
PYFUNCEBLE_DEBUG Tell us to log everything into the output/logs/*.log files.
PYFUNCEBLE_DEBUG_LVL Sets the logging level to use. --logging-level
PYFUNCEBLE_LOGGING_LVL Same as PYFUNCEBLE_DEBUG_LVL. --logging-level
PYFUNCEBLE_DEBUG_ON_SCREEN Tell us to log everything to stdout bool (true | false)
PYFUNCEBLE_CONFIG_DIR Tell us the location of the directory to use as the configuration directory.
PYFUNCEBLE_OUTPUT_LOCATION Tell us where we should generate the output/ directory.
APPDATA Used under Windows to construct/get the configuration directory if PYFUNCEBLE_CONFIG_DIR is not found.
GH_TOKEN Tell us the GitHub token to set into the repository configuration when using PyFunceble under Travis CI.
GL_TOKEN Tell us the GitLab token to set into the repository configuration when using PyFunceble under GitLab CI/CD.
GIT_EMAIL Tell us the git.email configuration to set when using PyFunceble under any supported CI environment.
GIT_NAME Tell us the git.name configuration to set when using PyFunceble under any supported CI environment.
TRAVIS_BUILD_DIR Used to confirm that we are running under a Travis CI container.
GITLAB_CI Used to confirm that we are running under a GitLab CI/CD environment.
GITLAB_USER_ID Used to confirm that we are running under a GitLab CI/CD environment.

Global overview

usage: pyfunceble [-d DOMAINS [DOMAINS ...]] [-u URLS [URLS ...]] [-f FILES [FILES ...]] [-uf URL_FILES [URL_FILES ...]] [--adblock] [--complements] [--preload] [--filter CLI_TESTING__FILE_FILTER] [--mining]
                [--rpz] [--wildcard] [-c] [--cooldown-time CLI_TESTING__COOLDOWN_TIME] [--local] [--dns-lookup] [--http] [--netinfo-lookup] [--special-lookup] [--whois-lookup] [--reputation-lookup]
                [--reputation] [--syntax] [-t LOOKUP__TIMEOUT] [-ua USER_AGENT__CUSTOM] [-vsc] [--dns DNS__SERVER [DNS__SERVER ...]] [--dns-protocol {UDP,TCP,HTTPS,TLS}] [--follow-server-order]
                [--trust-dns-server] [--inactive-db] [--database-type {csv,mariadb,mysql}] [-dbr CLI_TESTING__DAYS_BETWEEN__DB_RETEST] [-wdb] [-a] [-ex] [--colour]
                [--display-status {all,ACTIVE,INACTIVE,VALID,INVALID,MALICIOUS,SANE} [{all,ACTIVE,INACTIVE,VALID,INVALID,MALICIOUS,SANE} ...]] [--hierarchical] [-h] [-ip CLI_TESTING__HOSTS_IP] [--no-files]
                [--output-location OUTPUT_LOCATION] [--unified-results] [--percentage] [--plain] [--dots] [-q] [-s] [-w CLI_TESTING__MAX_WORKERS] [--ci-max-minutes CLI_TESTING__CI__MAX_EXEC_MINUTES] [--ci]
                [--ci-branch CLI_TESTING__CI__BRANCH] [--ci-distribution-branch CLI_TESTING__CI__DISTRIBUTION_BRANCH] [--ci-command CLI_TESTING__CI__COMMAND] [--ci-end-command CLI_TESTING__CI__END_COMMAND]
                [--ci-commit-message CLI_TESTING__CI__COMMIT_MESSAGE] [--ci-end-commit-message CLI_TESTING__CI__END_COMMIT_MESSAGE] [--help] [-v]

PyFunceble - The tool to check the availability or syntax of domain, IP or URL.

optional arguments:
    --help                Show this help message and exit.
    -v, --version         Show the version of PyFunceble and exit.

Test sources:
    -d DOMAINS [DOMAINS ...], --domain DOMAINS [DOMAINS ...]
                            Test one or more domains, separated by spaces.

                            When this option is used, no output files are generated.
    -u URLS [URLS ...], --url URLS [URLS ...]
                            Test one or more full URL, separated by spaces.
    -f FILES [FILES ...], --file FILES [FILES ...]
                            Read a local or remote (RAW link) file and test all domains inside it.
                            If remote (RAW link) file is given, PyFunceble will download it,
                            and test the content of the given RAW link as if it was a locally stored file.
    -uf URL_FILES [URL_FILES ...], --url-file URL_FILES [URL_FILES ...]
                            Read a local or remote (RAW link) file and test all (full) URLs inside it.
                            If remote (RAW link) file is given, PyFunceble will download it,
                            and test the content of the given RAW link as if it was a locally stored file.

                            This argument test if an URL is available. It ONLY test full URLs.

Source filtering, decoding, conversion and expansion:
    --adblock             Activates or deactivates the decoding of the adblock format.
                            Configured value: False
    --cidr                Activates or disables the expansion of CIDR formatted
                            addresses.
                            Configured value: False
    --complements         Activates or disables the generation and test of the
                            complements.
                            A complement is for example `example.org` if 'www.example.org'
                            is given and vice-versa.
                            Configured value: False
    --preload             Activates or disables the preloading of the input
                            file(s) into the continue dataset before starting the tests.

                            This reduces the waiting time while continuing a previous
                            session.
                            Note: This is useless when the auto continue subsystem is not active.
                            Configured value: False
    --filter CLI_TESTING__FILE_FILTER
                            Regex to match in order to test a given line.
                            Configured value: ''
    --mining              Activates or disables the mining subsystem.
                            Configured value: False
    --rpz                 Activates or disables the decoding of RPZ policies
                            from each given input files.
                            Configured value: False
    --wildcard            Activates or disables the decoding of wildcards for
                        each given input files.
                        Configured value: False

Test control:
    -c, --auto-continue, --continue
                            Activates or disables the autocontinue subsystem.
                            Configured value: False
    --cooldown-time CLI_TESTING__COOLDOWN_TIME
                            Sets the cooldown time (in second) to apply between
                            each test.
                            Configured value: 0.0
    --local               Activates or disables the consideration of the test(s)
                            in or for a local or private network context.
                            Configured value: False
    --dns-lookup          Activates or disables the usage of the DNS lookup
                            whether possible.
                            Configured value: True
    --http, --http-status-code-lookup
                            Switch the value of the usage of HTTP code.
                            Configured value: True
    --netinfo-lookup      Activates or disables the usage of the network
                            information (or network socket) whether possible.
                            Configured value: True
    --special-lookup      Activates or disables the usage of our SPECIAL and
                            extra rules whether possible.
                            Configured value: True
    --whois-lookup        Activates or disables the usage of the WHOIS record
                            (or better said the expiration date in it) whether possible.
                            Configured value: True
    --reputation-lookup   Activates or disables the usage of the reputation
                            dataset whether possible.
                            Configured value: False
    --reputation          Activates or disables the reputation checker.
                            Configured value: False
    --syntax              Activates or disables the syntax checker.
                            Configured value: False
    -t LOOKUP__TIMEOUT, --timeout LOOKUP__TIMEOUT
                            Sets the default timeout to apply to each lookup
                            utilities every time it is possible to define a timeout.
                            Configured value: 5
    -ua USER_AGENT__CUSTOM, --user-agent USER_AGENT__CUSTOM
                            Sets the user agent to use.

                            If not given, we try to get the latest (automatically) for you.
    -vsc, --verify-ssl-certificate
                            Activates or disables the verification of the SSL/TLS
                            certificate when testing for URL.
                            Configured value: False

DNS control:
    --dns DNS__SERVER [DNS__SERVER ...]
                            Sets one or more (space separated) DNS server(s) to use during testing.

                            To specify a port number for the DNS server you append
                            it as :port [ip:port].

                            If no port is specified, the default DNS port (53) is used.
                            Configured value: None
    --dns-protocol {UDP,TCP,HTTPS,TLS}
                            Sets the protocol to use for the DNS queries.
                            Configured value: 'UDP'
    --follow-server-order
                            Let us follow or mix the order of usage of the given DNS server(s).
                            Configured value: True
    --trust-dns-server    Activates or disable the trust mode.

                            When active, when the first read DNS server give us a negative response
                            - without error - we take it as it it.
                            Otherwise, if not active, when the first read DNS server give us
                            a negative response - without error - we still consolidate by
                            checking all given/found server.

                            Configured value: False
    --dns-delay DNS__DELAY
                            Sets the delay (in seconds) to apply between each DNS
                            queries.

                            Configured value: 0.0

Databases:
    --inactive-db         Activates or disables the usage of a 'database' to
                            store all 'INACTIVE' and 'INVALID'  subject for continuous retest.
                            Configured value: True
    --database-type {csv,mariadb,mysql}
                            Sets the database engine to use.
                            You can choose between the following: `csv | mariadb | mysql`
                            Configured value: 'csv'
    -dbr CLI_TESTING__DAYS_BETWEEN__DB_RETEST, --days-between-db-retest CLI_TESTING__DAYS_BETWEEN__DB_RETEST
                            Sets the numbers of days since the introduction of
                            subject into the inactive dataset before it gets retested.
                            Configured value: 1
    -wdb, --whois-database
                            Activates or disables the usage of a 'database' to
                            store the expiration date of all domains with a valid
                            expiration date.
                            Configured value: True

Output control:
    -a, --all             Activates or disables the display of the all
                            information in the table we print to stdout.
                            Configured value: False
    -ex, --execution      Activates or disables the display of the execution time.
                            Configured value: False
    --colour, --color     Activates or disables the coloration to STDOUT.
                            Configured value: True
    --display-status {all,ACTIVE,INACTIVE,VALID,INVALID,MALICIOUS,SANE} [{all,ACTIVE,INACTIVE,VALID,INVALID,MALICIOUS,SANE} ...]
                            Sets the status that we are allowed to print to STDOUT.

                            Multiple space separated statuses can be given.
                            Configured value: 'all'
    --hierarchical        Activates or disables the sorting of the files
                            content (output) in a hierarchical order.
                            Configured value: False
    -h, --host            Activates or disables the generation of the
                            hosts file(s).
                            Configured value: True
    -ip CLI_TESTING__HOSTS_IP, --hosts-ip CLI_TESTING__HOSTS_IP
                            Sets the IP to prefix each lines of the hosts file.
                            Configured value: '0.0.0.0'
    --no-files            Activates or disables the generation of any non-logs
                            file(s).
                            Configured value: False
    --output-location OUTPUT_LOCATION
                            Sets the location where we are supposed to generation
                            the output directory from.
                            Configured value: '/home/funilrys/repositories/github/source/PyFunceble'
    --unified-results     Activates or disables the generation of the unified
                            results file instead of the divided ones.
                            Configured value: False
    --percentage          Activates or disables the display and generation
                            of the percentage - file - of each status.
                            Configured value: True
    --plain               Activates or disables the generation of the
                            RAW file(s). What is meant is a list with only a list of
                            subject (one per line).
                            Configured value: False
    --dots                Activate or disables the display of dots or other
                            characters when we skip the test of a subject.
                            Configured value: False
    -q, --quiet           Activates or disables the display of output to the
                            terminal.
                            Configured value: False
    -s, --simple          Activates or disables the simple output mode.
                        Configured value: False

Multiprocessing:
    -w CLI_TESTING__MAX_WORKERS, --max-workers CLI_TESTING__MAX_WORKERS
                            Sets the number of maximal workers to use.
                            If not given, 40 (based on the current machine) will be applied.
                            Configured value: None

CI / CD:
    --ci-max-minutes CLI_TESTING__CI__MAX_EXEC_MINUTES
                            Sets the number of minutes to wait before starting
                            to stop a CI session.
                            Configured value: 15
    --ci                  Activates or disables the Continuous Integration
                            mechanism.
                            Configured value: False
    --ci-branch CLI_TESTING__CI__BRANCH
                            Sets our git working branch. This is the branch
                            from where we are supposed to store the tests
                            (excepts the final results).
                            Configured value: 'dev'
    --ci-distribution-branch CLI_TESTING__CI__DISTRIBUTION_BRANCH
                            Sets our git distributions branch. This is the
                            branch from where we are supposed to store and push
                            the final results.
                            Configured value: 'master'
    --ci-command CLI_TESTING__CI__COMMAND
                            Sets the command to execute before each commit
                            (except the final one).
                            Configured value: ''
    --ci-end-command CLI_TESTING__CI__END_COMMAND
                            Sets the command to execute before the final commit.
                            Configured value: ''
    --ci-commit-message CLI_TESTING__CI__COMMIT_MESSAGE
                            Sets the commit message to apply every time we have
                            to apply a commit except for the really last one.
                            Configured value: 'PyFunceble - AutoSave'
    --ci-end-commit-message CLI_TESTING__CI__END_COMMIT_MESSAGE
                            Sets the commit message to apply at the really end.
                            Configured value: 'PyFunceble - Results'

For an in-depth usage, explanation and examples of the arguments,
you should read the documentation at https://pyfunceble.readthedocs.io/en/latest/

Crafted with ♥ by Nissar Chababy (@funilrys) with the help of
https://git.io/JkUPS && https://git.io/JkUPF

Github Actions CI/CD

Example of how you can run PyFunceble inside Github Actions.

main.yml
name: PyFunceble CI tests
on:
push:
    branches:
    - "master"
pull_request:
    branches:
    - "master"
schedule:
    - cron: "0 * * * *"

env:
PYFUNCEBLE_AUTO_CONFIGURATION: "YES"
GIT_NAME: "${{ secrets.GIT_BOT_NAME }}"
GIT_EMAIL: "${{ secrets.GIT_BOT_EMAIL }}"
PYFUNCEBLE_CONFIG_DIR: "${{ github.workspace }}/.pyfunceble"
GITHUB_TOKEN: "${{ secrets.BOT_REPO_PAT }}"

jobs:
single:
    name: Run PyFunceble with a single domain
    runs-on: "${{ matrix.os }}"

    strategy:
    fail-fast: false
    matrix:
        python_version:
        - "3.9.1"
        os:
        - ubuntu-latest

    steps:
    - uses: actions/checkout@v2
        name: Clone repository
        with:
        token: "${{ secrets.BOT_REPO_PAT }}"

    - name: Set up Python ${{ matrix.python_version }}
        uses: actions/setup-python@v2
        with:
        python-version: ${{ matrix.python_version }}

    - name: Install dependencies
        run: |
        pip install --pre PyFunceble-dev

    - name: Get PyFunceble version
        run: |
        PyFunceble --version

    - name: Run PyFunceble
        run: |
        PyFunceble -a --logging-level critical -d github.com

file_and_push:
    name: Run PyFunceble against a file and push result to repository
    runs-on: "${{ matrix.os }}"

    strategy:
    fail-fast: false
    matrix:
        python_version:
        - "3.9.1"
        os:
        - ubuntu-latest

    steps:
    - uses: actions/checkout@v2
        name: Clone repository
        with:
        token: "${{ secrets.BOT_REPO_PAT }}"

    - name: Set up Python ${{ matrix.python_version }}
        uses: actions/setup-python@v2
        with:
        python-version: ${{ matrix.python_version }}

    - name: Install dependencies
        run: |
        pip install --pre PyFunceble-dev

    - name: Get PyFunceble version
        run: |
        PyFunceble --version

    - name: Run PyFunceble
        run: |
        PyFunceble -a --ci --logging-level critical -f test.list

GitLab Runner CI/CD

As we offer an argument named --ci which will autosave in a GitLab CI/CI environment, this document try to describe hot it works!

Configuration

Personal Access Token

A personal access token is needed in order for PyFunceble to automatically push the results.

You should get a personal GitLab access token with the read_repository and write_repository scopes.

Once created and copied in a safe place, create a new masked variable named GL_TOKEN inside the CI/CD settings of your project. The value of the variable should be the newly generated personal access token.

.gitlab-ci.yml

Note

This part only present a commented .gitlab-ci.yml. This is just an example do not take the following as necessarly true.

You’re invited to submit changes if something stated in this document is wrong.

# Python needed, so we use the python image.
image: python:latest

variables:
    # This is the Git name we have to set. (git config user.name)
    GIT_EMAIL: "dead-hosts@funilrys.com"
    # This is the Git Email we have to set. (git config user.email)
    GIT_NAME: "GitLab CI/CD"

before_script:
    # We install the development version of PyFunceble.
    # If you prefer the stable version replace `pyfunceble-dev`
    # with `pyfunceble`.
    - pip3 install PyFunceble-dev

run:
    script:
        # Let's say we want our results and our PyFunceble
        # infrastructure to be saved in a directory called `PyFunceble-tests`

        # We move inside it.
        - cd PyFunceble-tests
        # We test the file `my_awesome_list` which is located inside the current directory.
        # Note: we precise the `--ci` argument here,
        #     but you work without it if you set `ci: true` inside your `.PyFunceble.yaml`
        - PyFunceble --ci -f my_awesome_list --plain

Travis container CI/CD

As we offer an argument named --ci to activate the usage of PyFunceble in a Travis CI instance, we document here what you need to know!

Configuration

Note

This part only present a commented .travis.yml so that you can understand where to start.

If you need more practical examples, feel free to report to one of Dead-Hosts repositories which use PyFunceble with Travis CI.

env:
    global:
        # The following is your encrypted GitHub API key.
        # Indeed as we are going to push to the repository, this is needed.
        #- GH_TOKEN: # This can be set in the travis-ci https://travis-ci.com/repo/settings as 'Environment Variables'
        # or as below: secure: encrypted code
        - secure: QQdKFquFFojFT9XJ1XZp4EMoDTVoXFgqZq8XU+sCVf+pJQR6d/oKBp8rnSTCnZizWOQXUjGXUUxUpSG/dYGyBLjo3rH3rsn9ciZHVfubxbwK860w4sqibl4DvhCv2rdsFtvzXnhm4P9OL3i+krKdewh9fxpNyUU58qOgfnS7mK9FcFhb8z5ak2sxU2XRZedwm6Ro0oyVKs8kFkL4YaADfNyAHlGTfr9rVmE52WXQXQENktb9gFgR2A8ZnmLy0BCMZGkPDShJnjRDWD4DErtasLmLQvWpzOBwdbVJTY6U9KDRXVNdC9lp5E5Ba/dc0y36q6vjfgJR+QchetOtHgNbKYbLB8c26Di90OZCFJsxMNcl1Wct4qFPXkFGvjXrISW6pbdPL5Plto0Ig3iLiulhYOPVArysMIk9ymtSXP+WE7VWX01LQ1fEkIoSfeVZ2caTnCmTsoHVGRRe978CojKaT7yU45kb15hcyDrzptQ8EP2hfxeh5F7KtueQ6Rsb9LFDZMkMDKflZn6a+bRhESlmWWmYB9stzGzTurQA1E1bcSACJ8A8hG5nHBzZYJ2S+OY0PE7UdyOJ0JK0qe/67d+F9ocQdIoFpDDTdgIjHerQnD2wRg1aKPzLDb4jJTpqgr5ssPrqUAKl3st7gyaAZzCEADPDnIBDjOJS+mFWbx9DKgc=
        # This is the Git name we have to set. (git config user.name)
        - GIT_NAME: Travis CI
        # This is the Git Email we have to set. (git config user.email)
        - GIT_EMAIL: dead-hosts@funilrys.com

# This is the language we use.
language: python

# This is the python version we are going to use for the tests.
# Note: you can add any 3.x version to the list.
python:
- "3.8"

# The following will tell Travis CI to ends as fast as possible.
matrix:
    fast_finish: true

# Here we are setting what Travis CI have to cache.
cache:
    # We are caching pip3 as we use it to install PyFunceble
    - pip3

install:
    # We install the development version of PyFunceble. If you prefer the stable version replace
    # `pyfunceble-dev` with `pyfunceble`.
    - pip3 install pyfunceble-dev

# Our tests start here.
script:
    # Let's say we want our results and our PyFunceble infrastructure to be saved in a directory
    # called `PyFunceble-tests`

    # We move inside it.
    - cd PyFunceble-tests
    # We test the file `my_awesome_list` which is located inside the current directory.
    # Note: we precise the `--ci` argument here,
    #     but you work without it if you set `travis: true` inside your `.PyFunceble.yaml`
    - PyFunceble --ci -f my_awesome_list --plain

# The following initiate email notification logic.
notifications:
    # As we want to get a mail on failure and on status change, we set the following.
    on_success:   change
    on_failure:   always

Getting a GitHub token

For the secure index of the .travis.yml file, you have to generate a new GitHub token.

After you got your token, please write it or save it in a safe place as you’re going to need it every time you’re going to interact with Travis CI.

Note

The scope to set is public_repo but you can also set others depending on your needs.

Encrypting the token for future usage under the Travis CIs’ containers

To encrypt the token simply replace and execute the following according to your personal case.

$ travis encrypt 'GH_TOKEN=theGeneratedToken' -r 'The content of TRAVIS_REPO_SLUG' --add

Warning

Please do not execute the following explicitly without replacing theGeneratedToken with your previously generated GitHub token and The content of TRAVIS_REPO_SLUG with your repository slug.

Note

The usage of --add ensure that the travis program automatically add the secure index to the .travis.yml file.

Deprecated arguments

The following arguments have previous been in use by PyFunceble, these are now deprecated and should in some cases be replaced with alternative args.

In other cases the old feature have either been included into a other functional or simply been removed as it was obselete to keep the function within Pyfunceble v4.x.

This is happening because PyFunceble is evolving over time and features comes and goes.



-nw | --no-whois

Deprecated since version 4.0.0.

Replacement: –whois-lookup


--shadow-file | --shadow

Deprecated since version 4.0.0.

Replacement: N/A


--use-reputation-data

Deprecated since version 4.0.0.

Replacement: reputation-lookup


--dns-lookup-over-tcp

Deprecated since version 4.0.0.

See: –dns-protocol


-db | --database

Deprecated since version 4.0.0.

Replacement: –inactive-db


-dbc "something" | --days-between-db-clean

Deprecated since version 4.0.0.

Replacement: N/A


-json

Deprecated since version 4.0.0.

Replacement: N/A


-less

Deprecated since version 4.0.0.

Replacement: -a | –all


-nf

Deprecated since version 4.0.0.

Replacement: –no-files


-nl | --no-logs

Deprecated since version 4.0.0.

Replacement: –no-files


-nu | --no-unified

Deprecated since version 4.0.0.

Replacement: –unified-results


-ns|--no-special

Deprecated since version 4.0.0.

Replacement: –special-lookup

In the .code:.PyFunceble_production.yaml the value have changed from no_special to special


--split

Deprecated since version 4.0.0.

Replacement: –unified-results


--store-whois

Deprecated since version 4.0.0.

Replacement: N/A


-m | --multiprocess

Deprecated since version 4.0.0.

Integrated into -w | –max-workers


-p | --processes

Deprecated since version 4.0.0.

Replacement: -w | –max-workers


--multiprocess-merging-mode

Deprecated since version 4.0.0.

Replacement: N/A


--autosave-minutes

Deprecated since version 4.0.0.

Replacement: –ci-max-minutes


--cmd

Deprecated since version 4.0.0.

Replacement: –ci-command


--cmd-before-end

Deprecated since version 4.0.0.

Replacement: --ci-end-command