Usage¶

From a terminal¶

`--help`¶

Show the help message and exit.

`-v` | `--version`¶

Show the version of PyFunceble and exit.

Source¶

`-d "something"` | `--domain "something"`¶

Test one or more domains, separated by spaces.

Note

This argument takes one or more values.

As example:

$ PyFunceble -d example.org example.net

Note

When this option is used, no output files are generated.

`-url "something"` | `--url "something"`¶

Test one or more full URL, separated by spaces.

Note

When we test the availability of a URL, we (only) check the HTTP status code of the given URL.

Note

This argument takes one or more values.

As example:

$ PyFunceble -u https://example.org https://example.com

`-f "something"` | `--file "something"`¶

Read a local or remote (RAW link) file and test all domains inside it. If remote (RAW link) file is given, PyFunceble will download it, and test the content of the given RAW link as if it was a locally stored file.

Multiple space separated files can be given.

Note

This argument takes one or more values.

As example:

$ PyFunceble -f test_this test_that

`-uf "something"` | `--url-file "something"`¶

Read a local or remote (RAW link) file and test all (full) URLs inside it. If remote (RAW link) file is given, PyFunceble will download it, and test the content of the given RAW link as if it was a locally stored file.

Note

This argument takes one or more values.

As example:

$ PyFunceble -uf test_this test_tha

Note

We consider one line as one URL to test.

Note

This argument test if a URL which is inside the given file is available. It ONLY tests full URLs.

As example:

$ PyFunceble -uf `https://raw.githubusercontent.com/funilrys/PyFunceble/dev/.travis/lists/url`

will download the given URL and test for its content assuming that each line represents a URL to test.

Warning

A test with this argument consists of the comparison of the status code. No WHOIS record will be requested nor DNS Lookup will be done.

Source filtering, decoding, conversion and expansion¶

`--adblock`¶

Activates or disables the decoding of the adblock format.

Default value: False

Note

If this argument is activated the system will extract all domains or IP from the given adblock file.

`--complements`¶

Activates or disables the generation and test of the complements. A complement is for example example.org if www.example.org is given and vice-versa.

Default value: False

`--filter "something"`¶

Regex to match in order to test a given line.

Default value: None

Want to test all blogspot from your list? This argument allows you to do that!

Note

This argument should be a regex expression.

`--mining`¶

Activates or disables the mining subsystem usage.

Default value: False

Want to find domain or URL linked to a domain in your list? This argument will exactly do that.

`--rpz`¶

Activates or disables the decoding of RPZ policies from each given input files.

Default value: False

`--wildcard`¶

Activates or disables the decoding of wildcards for each given input files.

Default value: False

Test control¶

`-c` | `--auto-continue` | `--continue`¶

Activates or disables the autocontinue subsystem.

Default value: True

This argument activates or deactivates the auto-continue subsystem. Indeed, as we can automatically continue if the script has been stopped, this switch allows us to disable or enable the usage of that specific subsystem.

`--cooldown-time`¶

Sets the cooldown time (in second) to apply between each test.

Default value: 0.0

This argument applies a number of seconds to sleep before/between each test.

`--local`¶

Activates or disables the consideration of the test(s) in or for a local or private network context.

Default value: False

Want to run a test over a local or private network? This argument will disable the limitation which does not apply to private networks.

`--dns-lookup`¶

Activates or disables the usage of the DNS lookup whether possible.

Default value: True

Don’t want to perform some DNS lookup ? This argument is for you.

`--http-status-code-lookup` | `--http`¶

Activates or disables the usage of the HTTP status code whether possible.

Default value: True

Don’t want to take the result of the HTTP code execution into consideration? This argument allows you to disable that!

`--netinfo-lookup`¶

Activates or disables the usage of the network information (or network socket) whether possible.

Default value: True

Don’t want to perform some netinfo lookup ? This argument is for you.

`--special-lookup`¶

Activates or disables the usage of our SPECIAL and extra rules whether possible.

Default value: True

Don’t want to use/apply the SPECIAL rules - which are explained in the source column section? This argument disables them all.

`--whois-lookup`¶

Activates or disables the usage of the WHOIS record (or better said the expiration date in it) whether possible.

Default value: True

Don’t want to use or take into consideration the results from whois? This argument allows you to disable it!

`--reputation-lookup`¶

Activates or disables the usage of the reputation dataset whether possible.

Default value: False

Want to take the reputation data into consideration? This argument is for you.

`--reputation`¶

Activates or disables the reputation checker.

Default value: False

`--syntax`¶

Activates or disables the syntax checker.

Default value: False

`-t "something"` | `--timeout "something"`¶

Sets the default timeout to apply to each lookup utilities everytime it is possible to define a timeout.

Default value: 5

`-ua "something"` | `--user-agent "something"`¶

Sets the user agent to use.

Warning

If not given, we try to get the latest (automatically) for you

`-vsc` | `--verify-ssl-certificate`¶

Activates or disables the verification of the SSL/TLS certificate when testing for URL.

Default value: False

Warning

If you activate the verification of the SSL/TLS certificate, you may get false-positive results.

Indeed if the certificate is not registered to the CA or is simply invalid and the domain is still alive, you will always get INACTIVE as output.

DNS control¶

`--dns`¶

Sets one or more DNS server(s) to use during testing. Separated by spaces.

Default value: Follow OS DNS ==> None

Warning

We expect a DNS server(s). If no DNS server(s) is given. You’ll almost for certain get all results as INACTIVE

This could happen in case you use --dns -f

Note

You can specify a port number to use to the DNS server if needed.

As example:

- 127.0.1.53:5353

`--dns-protocol`¶

Sets the protocol to use for the DNS queries.

Default value: False

Available values: UDP, TCP, HTTPS, TLS.

Databases¶

`--inactive-database`¶

Switch the value of the usage of a database to store inactive domains of the currently tested list.

Default value: True

This argument will disable or enable the usage of a database which saves all INACTIVE and INVALID domain of the given file over time.

`--database-type`¶

Sets the database engine to use.

Default value: csv

Available values: csv, mariadb, mysql.

`-dbr "something"` | `--days-between-db-retest "something"`¶

Sets the numbers of days since the introduction of a subject into the inactive dataset before it gets retested.

Default value: 1

Note

This argument is only used if -db or inactive_database : true (under .PyFunceble.yaml) are activated.

`-wdb` | `--whois-database`¶

Activates or disables the uage of a “database” to store the expiration date of all domains with a valid expiration date.

Default value: True

Output control¶

`-a` | `--all`¶

Activates or disables the disply of the all information in the table we print to stdout.

Default value: False

When activated:

Domain                        Status      Expiration Date   Source     HTTP Code  Checker
----------------------------- ----------- ----------------- ---------- ---------- -------------
pyfunceble.readthedocs.io     ACTIVE      Unknown           NSLOOKUP   302        AVAILABILITY

When deactivated:

Domain                        Status      Source
----------------------------- ----------- ----------
pyfunceble.readthedocs.io     ACTIVE      SYNTAX

`-ex` | `--execution`¶

Activates or disables the display of the execution time.

Default value: False

Want to know the execution time of your test? Well, this argument will let you know!

`--color` | `--colour`¶

Activates or disables the coloration to STDOUT.

Default value: True

Don’t want any colour ? This argument is for you!

`--display-status`¶

Sets the status that we are allowed to print to STDOUT. Multiple space separated statuses can be given.

Default value: all

Available values: all, ACTIVE, INACTIVE, INVALID, VALID, SANE, MALICIOUS

`--hierarchical`¶

Activates or disables the sorting of the files content (output) in a hierarchical order.

Default value: True

This argument will output the result listed in a hierarchical order.

`-h` | `--host`¶

Activates or disables the generation of the hosts file(s).

Default value: True

This argument will let the system know if it has to generate the hosts file version of each status.

`-ip "something"` | –hosts-ip “something”¶

Sets the IP to prefix each lines of the hosts file.

Default value: 0.0.0.0

`--no-files`¶

Activates or disables the generation of any non-logs file(s).

Default value: False

Want to disable the production of the outputted files? This argument is for you!

`--output-location`¶

Sets the location where we are supposed to generation the output directory from.

Default value: Please look at your machine

Don’t want to use the PYFUNCEBLE_OUTPUT_DIR environment variable? This argument is for you!

`--unified-results`¶

Activates or disables the generation of the unified results file instead of the splitted one.

Default value: True

This argument disables the generation of the result.txt file.

`--percentage`¶

Activates or disables the display and generation of the percentage - file - of each status.

Default value: True

This argument will disable or enable the generation of the percentage of each status.

`--plain`¶

Activates or disables the generation of the RAW file(s). What is meant is a list with only a list of subject (one per line).

Default value: False:

Want to get a list with all domains for each status? The activation of this argument does the work while testing!

`--dots`¶

Activate or disables the display of dots or other characters when we skip the test of a subjec.

Default value: False

`-q` | `--quiet`¶

Activates or disables the display of output to the terminal.

Default value: False

You prefer to run a program silently? This argument is for you!

`-s` | `--simple`¶

Activates or disables the simple output mode.

Default value: False

Want as less as possible data on screen? This argument returns as less as possible on screen!

Multithreading¶

`-w` | `--max-workers`¶

Sets the number of maximal worker to use.

Default value: False

Note

If omitted, the number of available CPU cores multiplied by 5 will be used instead.

CI / CD¶

`--ci-max-minutes`¶

Sets the number of minutes to wait before starting to stop a CI session.

Default value: 15

`--ci`¶

Activates or disables the Continuous Integration mechanism.

Default value: False

Note

If you combine this argument with the --quiet argument, the test will output a dotted line, where each dot (.) represent one test result or input which was skipped because it was previously tested.

Want to use PyFunceble under a supported CI infrastructure/network? This argument is suited for your needs!

`--ci-branch`¶

Sets our git working branch. This is the branch from where we are supposed to store the tests (excepts the final results).

Default value: master

Note

Currently the branch need to exist, but there are being worked on a path to have PyFunceble to create the sub-branch and finally merge it into the --ci-distribution-branch

`--ci-distribution-branch`¶

Sets our git distributions branch. This is the branch from where we are supposed to store and push the final results.

Default value: master

Note

The difference between this and --ci-branch is the fact that this branch will get the (final) result only when the test is finished under the given --ci-branch.

As an example, this allows us to have 2 branches:

proceessing (CI branch), for the tests with PyFunceble.
master (CI distribution branch), for the distribution of the results of PyFunceble.

`--ci-command "something"` | `--cmd "something"`¶

Sets the command to execute before each commit (except the final one).

Default value: ''

Note

In this example, something should be a script or a program which have to be executed when we reached the end of the given file.

Note

This argument is only used if --ci or ci: true (under .PyFunceble.yaml) are activated.

`--ci-end-command "something"` | `--cmd-before-end "something"`¶

Sets the command to execute before the final commit.

Default value: ''

Note

In this example, something should be a script or a program which have to be executed when we reached the end of the given file.

Note

This argument is only used if --ci or ci: true (under .PyFunceble.yaml) are activated.

`--ci-commit-message "something"` | `--commit-autosave-message "something"`¶

Sets the commit message to apply everytime we have to apply a commit except for the really last one.

Default value: PyFunceble - AutoSave

This argument allows us to set a custom commit message which is going to be used as a commit message when saving.

Note

This argument is only used if --ci or ci: true (under .PyFunceble.yaml) are used.

Note

This argument is only used if we have to split the work into multiple processes because a list is too long or the timeout is reached.

Warning

Please avoid the usage of [ci skip] here.

`--ci-end-commit-message "something"` | `--commit-results-message "something"`¶

Sets the commit message to apply at the really end.

Default value: PyFunceble - Results

Note

This argument is only used if --ci or ci: true (under .PyFunceble.yaml) are used.

Note

This argument is only used if we reached the end of the list we are or have to test.

Global overview¶

usage: pyfunceble [-d DOMAINS [DOMAINS ...]] [-u URLS [URLS ...]]
                [-f FILES [FILES ...]] [-uf URL_FILES [URL_FILES ...]]
                [--adblock] [--complements]
                [--filter CLI_TESTING__FILE_FILTER] [--mining] [--rpz]
                [--wildcard] [-c]
                [--cooldown-time CLI_TESTING__COOLDOWN_TIME] [--local]
                [--dns-lookup] [--http] [--netinfo-lookup]
                [--special-lookup] [--whois-lookup] [--reputation-lookup]
                [--reputation] [--syntax] [-t LOOKUP__TIMEOUT]
                [-ua USER_AGENT__CUSTOM] [-vsc]
                [--dns DNS__SERVER [DNS__SERVER ...]]
                [--dns-protocol {UDP,TCP,HTTPS,TLS}] [--inactive-db]
                [--database-type {csv,mariadb,mysql}]
                [-dbr CLI_TESTING__DAYS_BETWEEN__DB_RETEST]
                [-wdb CLI_TESTING__WHOIS_DB] [-a] [-ex] [--colour]
                [--display-status {all,ACTIVE,INACTIVE,VALID,INVALID,MALICIOUS,SANE} [{all,ACTIVE,INACTIVE,VALID,INVALID,MALICIOUS,SANE} ...]]
                [--hierarchical] [-h] [-ip CLI_TESTING__HOSTS_IP]
                [--no-files] [--output-location OUTPUT_LOCATION]
                [--unified-results] [--percentage] [--plain] [--dots] [-q]
                [-s] [-w CLI_TESTING__MAX_WORKERS]
                [--ci-max-minutes CLI_TESTING__CI__MAX_EXEC_MINUTES] [--ci]
                [--ci-branch CLI_TESTING__CI__BRANCH]
                [--ci-distribution-branch CLI_TESTING__CI__DISTRIBUTION_BRANCH]
                [--cmd CLI_TESTING__CI__COMMAND]
                [--cmd-before-end CLI_TESTING__CI__END_COMMAND]
                [--ci-commit-message CLI_TESTING__CI__COMMIT_MESSAGE]
                [--ci-end-commit-message CLI_TESTING__CI__END_COMMIT_MESSAGE]
                [--help] [-v]

PyFunceble - The tool to check the availability or syntax of domain, IP or URL.

optional arguments:
    --help                Show this help message and exit.
    -v, --version         Show the version of PyFunceble and exit.

Source:
    -d DOMAINS [DOMAINS ...], --domain DOMAINS [DOMAINS ...]
                            Test one or more domains, separated by spaces.

                            When this option is used, no output files are generated.
    -u URLS [URLS ...], --url URLS [URLS ...]
                            Test one or more full URL, separated by spaces.
    -f FILES [FILES ...], --file FILES [FILES ...]
                            Read a local or remote (RAW link) file and test all domains inside it.
                            If remote (RAW link) file is given, PyFunceble will download it,
                            and test the content of the given RAW link as if it was a locally stored file.
    -uf URL_FILES [URL_FILES ...], --url-file URL_FILES [URL_FILES ...]
                            Read a local or remote (RAW link) file and test all (full) URLs inside it.
                            If remote (RAW link) file is given, PyFunceble will download it,
                            and test the content of the given RAW link as if it was a locally stored file.

                            This argument test if an URL is available. It ONLY test full URLs.

Source filtering, decoding, conversion and expansion:
    --adblock             Activates or deactivates the decoding of the adblock format.
                            Configured value: False
    --complements         Activates or disables the generation and test of the
                            complements.
                            A complement is for example `example.org` if 'www.example.org'
                            is given and vice-versa.
                            Configured value: False
    --filter CLI_TESTING__FILE_FILTER
                            Regex to match in order to test a given line.
                            Configured value: None
    --mining              Activates or disables the mining subsystem.
                            Configured value: False
    --rpz                 Activates or disables the decoding of RPZ policies
                            from each given input files.
                            Configured value: False
    --wildcard            Activates or disables the decoding of wildcards for
                            each given input files.
                            Configured value: False

Test control:
    -c, --auto-continue, --continue
                            Activates or disables the autocontinue subsystem.
                            Configured value: True
    --cooldown-time CLI_TESTING__COOLDOWN_TIME
                            Sets the cooldown time (in second) to apply between
                            each test.
                            Configured value: 0.0
    --local               Activates or disables the consideration of the test(s)
                            in or for a local or private network context.
                            Configured value: False
    --dns-lookup          Activates or disables the usage of the DNS lookup
                            whether possible.
                            Configured value: True
    --http, --http-status-code-lookup
                            Switch the value of the usage of HTTP code.
                            Configured value: True
    --netinfo-lookup      Activates or disables the usage of the network
                            information (or network socket) whether possible.
                            Configured value: True
    --special-lookup      Activates or disables the usage of our SPECIAL and
                            extra rules whether possible.
                            Configured value: True
    --whois-lookup        Activates or disables the usage of the WHOIS record
                            (or better said the expiration date in it) whether possible.
                            Configured value: True
    --reputation-lookup   Activates or disables the usage of the reputation
                            dataset whether possible.
                            Configured value: False
    --reputation          Activates or disables the reputation checker.
                            Configured value: False
    --syntax              Activates or disables the syntax checker.
                            Configured value: False
    -t LOOKUP__TIMEOUT, --timeout LOOKUP__TIMEOUT
                            Sets the default timeout to apply to each lookup
                            utilities everytime it is possible to define a timeout.
                            Configured value: 5
    -ua USER_AGENT__CUSTOM, --user-agent USER_AGENT__CUSTOM
                            Sets the user agent to use.

                            If not given, we try to get the lastest (automatically) for you.
    -vsc, --verify-ssl-certificate
                            Activates or disables the verification of the SSL/TLS
                            certificate when testing for URL.
                            Configured value: False

DNS control:
    --dns DNS__SERVER [DNS__SERVER ...]
                            Sets one or more (space separated) DNS server(s) to use during testing.

                            To specify a port number for the DNS server you append
                            it as :port [ip:port].

                            If no port is specified, the default DNS port (53) is used.
                            Configured value: None
    --dns-protocol {UDP,TCP,HTTPS,TLS}
                            Sets the protocol to use for the DNS queries.
                            Configured value: 'UDP'

Databases:
    --inactive-db         Activates or disables the usage of a 'database' to
                            store all 'INACTIVE' and 'INVALID'  subject for continuous retest.
                            Configured value: True
    --database-type {csv,mariadb,mysql}
                            Sets the database engine to use.
                            You can choose between the following: `csv | mariadb | mysql`
                            Configured value: 'csv'
    -dbr CLI_TESTING__DAYS_BETWEEN__DB_RETEST, --days-between-db-retest CLI_TESTING__DAYS_BETWEEN__DB_RETEST
                            Sets the numbers of days since the introduction of
                            subject into the inactive dataset before it gets retested.
                            Configured value: 1
    -wdb CLI_TESTING__WHOIS_DB, --whois-database CLI_TESTING__WHOIS_DB
                            Activates or disables the usage of a 'database' to
                            store the expiration date of all domains with a valid
                            expiration date.
                            Configured value: True

    Output control:
    -a, --all             Activates or disables the display of the all
                            information in the table we print to stdout.
                            Configured value: False
    -ex, --execution      Activates or disables the display of the execution time.
                            Configured value: False
    --colour, --color     Activates or disables the coloration to STDOUT.
                            Configured value: True
    --display-status {all,ACTIVE,INACTIVE,VALID,INVALID,MALICIOUS,SANE} [{all,ACTIVE,INACTIVE,VALID,INVALID,MALICIOUS,SANE} ...]
                            Sets the status that we are allowed to print to STDOUT.

                            Multiple space separated statuses can be given.
                            Configured value: 'all'
    --hierarchical        Activates or disables the sorting of the files
                            content (output) in a hierarchical order.
                            Configured value: False
    -h, --host            Activates or disables the generation of the
                            hosts file(s).
                            Configured value: True
    -ip CLI_TESTING__HOSTS_IP, --hosts-ip CLI_TESTING__HOSTS_IP
                            Sets the IP to prefix each lines of the hosts file.
                            Configured value: '0.0.0.0'
    --no-files            Activates or disables the generation of any non-logs
                            file(s).
                            Configured value: False
    --output-location OUTPUT_LOCATION
                            Sets the location where we are supposed to generation
                            the output directory from.
                            Configured value: '/XXXX/XXXX/XXXXX'
    --unified-results     Activates or disables the generation of the unified
                            results file instead of the splitted one.
                            Configured value: False
    --percentage          Activates or disables the display and generation
                            of the percentage - file - of each status.
                            Configured value: True
    --plain               Activates or disables the generation of the
                            RAW file(s). What is meant is a list with only a list of
                            subject (one per line).
                            Configured value: True
    --dots                Activate or disables the display of dots or other
                            characters when we skip the test of a subject.
                            Configured value: False
    -q, --quiet           Activates or disables the display of output to the
                            terminal.
                            Configured value: False
    -s, --simple          Activates or disables the simple output mode.
                            Configured value: False

Multithreading:
    -w CLI_TESTING__MAX_WORKERS, --max-workers CLI_TESTING__MAX_WORKERS
                            Sets the number of maximal workers to use.
                            If not given, 40 (based on the current machine) will be applied.
                            Configured value: None

CI / CD:
    --ci-max-minutes CLI_TESTING__CI__MAX_EXEC_MINUTES, --autosave-minutes CLI_TESTING__CI__MAX_EXEC_MINUTES
                            Sets the number of minutes to wait before starting
                            to stop a CI session.
                            Configured value: 15
    --ci                  Activates or disables the Continuous Integration
                            mechanism.
                            Configured value: False
    --ci-branch CLI_TESTING__CI__BRANCH
                            Sets our git working branch. This is the branch
                            from where we are supposed to store the tests
                            (excepts the final results).
                            Configured value: 'master'
    --ci-distribution-branch CLI_TESTING__CI__DISTRIBUTION_BRANCH
                            Sets our git distributions branch. This is the
                            branch from where we are supposed to store and push
                            the final results.
                            Configured value: 'master'
    --cmd CLI_TESTING__CI__COMMAND, --ci-command CLI_TESTING__CI__COMMAND
                            Sets the command to execute before each commit
                            (except the final one).
                            Configured value: None
    --cmd-before-end CLI_TESTING__CI__END_COMMAND, --ci-end-command CLI_TESTING__CI__END_COMMAND
                            Sets the command to execute before the final commit.
                            Configured value: None
    --ci-commit-message CLI_TESTING__CI__COMMIT_MESSAGE, --commit-autosave-message CLI_TESTING__CI__COMMIT_MESSAGE
                            Sets the commit message to apply everytime we have
                            to apply a commit except for the really last one.
                            Configured value: 'PyFunceble - AutoSave'
    --ci-end-commit-message CLI_TESTING__CI__END_COMMIT_MESSAGE, --commit-results-message CLI_TESTING__CI__END_COMMIT_MESSAGE
                            Sets the commit message to apply at the really end.
                            Configured value: 'PyFunceble - Results'

For an in-depth usage, explanation and examples of the arguments,
you should read the documentation at https://pyfunceble.readthedocs.io/en/dev/

Crafted with ♥ by Nissar Chababy (@funilrys) with the help of
https://git.io/JkUPS && https://git.io/JkUPF

From a GitLab CI/CD environment¶

As we offer an argument named --ci which will autosave in a GitLab CI/CI environment, this document try to describe hot it works!

Configuration¶

Personal Access Token¶

A personal access token is needed in order for PyFunceble to automatically push the results.

You should get a personal GitLab access token with the read_repository and write_repository scopes.

Once created and copied in a safe place, create a new masked variable named GL_TOKEN inside the CI/CD settings of your project. The value of the variable should be the newly generated personal access token.

`.gitlab-ci.yml`¶

Note

This part only present a commented .gitlab-ci.yml. This is just an example do not take the following as necessarly true.

You’re invited to submit changes if something stated in this document is wrong.

# Python needed, so we use the python image.
image: python:latest

variables:
    # This is the Git name we have to set. (git config user.name)
    GIT_EMAIL: "dead-hosts@funilrys.com"
    # This is the Git Email we have to set. (git config user.email)
    GIT_NAME: "GitLab CI/CD"

before_script:
    # We install the development version of PyFunceble.
    # If you prefer the stable version replace `pyfunceble-dev`
    # with `pyfunceble`.
    - pip3 install PyFunceble-dev

run:
    script:
        # Let's say we want our results and our PyFunceble
        # infrastructure to be saved in a directory called `PyFunceble-tests`

        # We move inside it.
        - cd PyFunceble-tests
        # We test the file `my_awesome_list` which is located inside the current directory.
        # Note: we precise the `--ci` argument here,
        #     but you work without it if you set `ci: true` inside your `.PyFunceble.yaml`
        - PyFunceble --ci -f my_awesome_list --plain

From a Travis CI container¶

As we offer an argument named --ci to activate the usage of PyFunceble in a Travis CI instance, we document here what you need to know!

Configuration¶

Note

This part only present a commented .travis.yml so that you can understand where to start.

If you need more practical examples, feel free to report to one of Dead-Hosts repositories which use PyFunceble with Travis CI.

env:
    global:
        # The following is your encrypted GitHub API key.
        # Indeed as we are going to push to the repository, this is needed.
        #- GH_TOKEN: # This can be set in the travis-ci https://travis-ci.com/repo/settings as 'Environment Variables'
        # or as below: secure: encrypted code
        - secure: QQdKFquFFojFT9XJ1XZp4EMoDTVoXFgqZq8XU+sCVf+pJQR6d/oKBp8rnSTCnZizWOQXUjGXUUxUpSG/dYGyBLjo3rH3rsn9ciZHVfubxbwK860w4sqibl4DvhCv2rdsFtvzXnhm4P9OL3i+krKdewh9fxpNyUU58qOgfnS7mK9FcFhb8z5ak2sxU2XRZedwm6Ro0oyVKs8kFkL4YaADfNyAHlGTfr9rVmE52WXQXQENktb9gFgR2A8ZnmLy0BCMZGkPDShJnjRDWD4DErtasLmLQvWpzOBwdbVJTY6U9KDRXVNdC9lp5E5Ba/dc0y36q6vjfgJR+QchetOtHgNbKYbLB8c26Di90OZCFJsxMNcl1Wct4qFPXkFGvjXrISW6pbdPL5Plto0Ig3iLiulhYOPVArysMIk9ymtSXP+WE7VWX01LQ1fEkIoSfeVZ2caTnCmTsoHVGRRe978CojKaT7yU45kb15hcyDrzptQ8EP2hfxeh5F7KtueQ6Rsb9LFDZMkMDKflZn6a+bRhESlmWWmYB9stzGzTurQA1E1bcSACJ8A8hG5nHBzZYJ2S+OY0PE7UdyOJ0JK0qe/67d+F9ocQdIoFpDDTdgIjHerQnD2wRg1aKPzLDb4jJTpqgr5ssPrqUAKl3st7gyaAZzCEADPDnIBDjOJS+mFWbx9DKgc=
        # This is the Git name we have to set. (git config user.name)
        - GIT_NAME: Travis CI
        # This is the Git Email we have to set. (git config user.email)
        - GIT_EMAIL: dead-hosts@funilrys.com

# This is the language we use.
language: python

# This is the python version we are going to use for the tests.
# Note: you can add any 3.x version to the list.
python:
- "3.8"

# The following will tell Travis CI to ends as fast as possible.
matrix:
    fast_finish: true

# Here we are setting what Travis CI have to cache.
cache:
    # We are caching pip3 as we use it to install PyFunceble
    - pip3

install:
    # We install the development version of PyFunceble. If you prefer the stable version replace
    # `pyfunceble-dev` with `pyfunceble`.
    - pip3 install pyfunceble-dev

# Our tests start here.
script:
    # Let's say we want our results and our PyFunceble infrastructure to be saved in a directory
    # called `PyFunceble-tests`

    # We move inside it.
    - cd PyFunceble-tests
    # We test the file `my_awesome_list` which is located inside the current directory.
    # Note: we precise the `--ci` argument here,
    #     but you work without it if you set `travis: true` inside your `.PyFunceble.yaml`
    - PyFunceble --ci -f my_awesome_list --plain

# The following initiate email notification logic.
notifications:
    # As we want to get a mail on failure and on status change, we set the following.
    on_success:   change
    on_failure:   always

Getting a GitHub token¶

For the secure index of the .travis.yml file, you have to generate a new GitHub token.

After you got your token, please write it or save it in a safe place as you’re going to need it every time you’re going to interact with Travis CI.

Note

The scope to set is public_repo but you can also set others depending on your needs.

Encrypting the token for future usage under the Travis CIs’ containers¶

To encrypt the token simply replace and execute the following according to your personal case.

$ travis encrypt 'GH_TOKEN=theGeneratedToken' -r 'The content of TRAVIS_REPO_SLUG' --add

Warning

Please do not execute the following explicitly without replacing theGeneratedToken with your previously generated GitHub token and The content of TRAVIS_REPO_SLUG with your repository slug.

Note

The usage of --add ensure that the travis program automatically add the secure index to the .travis.yml file.

Using the PyFunceble (Python) API¶

If you are working with a python script, module or even class, you can integrate PyFunceble to your main logic by importing it and using its API (cf: API Documentation).

This section will present some example of the way you can interact with PyFunceble from anything written in Python.

Get the availability of domains or IP¶

Note

This example can be found in our examples repository.

Todo

Add IPs in the loop.

"""
This is an example which respond to the following problematic(s):

    * How can I get the avaibility of a domain or IP with PyFunceble ?
"""

# We want some coloration so we import the tool do to that :)
from PyFunceble import initiate_colorama, Fore, Style
# We import the tool to print the colored CLI logo.
from PyFunceble.cli_core import CLICore
# We import the configuration loader.
from PyFunceble import load_config
# We import the test method of the PyFunceble API.
from PyFunceble import test as PyFunceble

# We initiate the list of domains we are going to test.
DOMAINS = [
    "google.com",
    "tweeetttter.com",
    "github.com",
    "examplessss.ooooorgg",
    "twitter.com",
    "forest-jump"
]

# We initiate colorama.
initiate_colorama(True)

# We load our configuration.
#
# Note: We need this to print the logo but if you
# doesn't need the logo, you can ignore this.
load_config(generate_directory_structure=False)

# We print the PyFunceble logo.
CLICore.colorify_logo(home=True)

def print_result(subject, status):
    """
    Given the subject and its status, we print it to STDOUT.

    :param str subject: The subject we are going to print.
    :param str status: The status of the domain.
    """

    if status == "ACTIVE":
        print(f"{Fore.GREEN + Style.BRIGHT}{domain} is {status}")
    elif status == "INACTIVE":
        print(f"{Fore.RED + Style.BRIGHT}{domain} is {status}")
    else:
        print(f"{Fore.CYAN + Style.BRIGHT}{domain} is {status}")

for domain in DOMAINS:
    # We loop through the list of domain.

    # And we print the domain and status with the right coloration!
    print_result(domain, PyFunceble(domain))

Get the availability of URL¶

Note

This example can be found in our examples repository.

"""
This is an example which respond to the following problematic(s):

    * How can I get the avaibility of an URL with PyFunceble ?
"""

# We want some coloration so we import the tool do to that :)
from PyFunceble import initiate_colorama, Fore, Style
# We import the tool to print the colored CLI logo.
from PyFunceble.cli_core import CLICore
# We import the configuration loader.
from PyFunceble import load_config
# We import the test method of the PyFunceble API.
from PyFunceble import url_test as PyFunceble

# We initiate the list of URLs we are going to test.
URLS = [
    "https://google.com",
    "http://tweeetttter.com",
    "ftp://github.com",
    "http://examplessss.ooooorgg",
    "http://twitter.com",
]

# We initiate colorama.
initiate_colorama(True)

# We load our configuration.
#
# Note: We need this to print the logo but if you
# doesn't need the logo, you can ignore this.
load_config(generate_directory_structure=False)

# We print the PyFunceble logo.
CLICore.colorify_logo(home=True)

def print_result(subject, status):
    """
    Given the subject and its status, we print it to STDOUT.

    :param str subject: The subject we are going to print.
    :param str status: The status of the domain.
    """

    if status == "ACTIVE":
        print(f"{Fore.GREEN + Style.BRIGHT}{domain} is {status}")
    elif status == "INACTIVE":
        print(f"{Fore.RED + Style.BRIGHT}{domain} is {status}")
    else:
        print(f"{Fore.CYAN + Style.BRIGHT}{domain} is {status}")

for url in URLS:
    # We loop through the list of domain.

    # And we print the domain and status with the right coloration!
    print_result(url, PyFunceble(url))

Complete dataset while getting the avaibility of domains, IPs or URL¶

While using our API, you can request to see/get everything with the help of the complete=True argument.

You’ll then get the following dict as output.

{
    "_status": None, # If some extra rules are applied, this index will keep the status before the extra rules was applied.
    "_status_source": None, # If some extra rules are applied, this index will keep the source before the extra rules was applied.
    "domain_syntax_validation": None, # The domain syntax validation status.
    "expiration_date": None, # The expiration date of the tested subject (if found).
    "http_status_code": None, # The status code of the tested subejct.
    "ip4_syntax_validation": None, # The IPv4 syntax validation status.
    "dns_lookup": [], # The DNS Lookup output.
    "status_source": None, # The (final) source which gave us the status.
    "status": None, # The (final) status returned by PyFunceble.
    "tested": None, # The tested subject.
    "url_syntax_validation": None, # The url syntax validation status.
    "whois_record": None, # The whois record (if found).
    "whois_server": None, # The whois server we use to get the whois record (if found).
}

Set custom configuration index while getting the avaibility of domains, IPs or URL¶

While using PyFunceble, you might want to set or overwritte a default behaviour.

You can do that in 2 ways. Globally or locally.

Globally¶

To set globally simply initiate the configuration loader and parse your custom configuration along with the initialization.

As example, you can do it like follow:

# We import the configuration loader.
from PyFunceble import load_config

# We set our list of indexes to overwritte.
OUR_PYFUNCEBLE_CONFIG = {"share_logs":False, "no_files": True}

# We load our configuration and parse our custom indexes.
load_config(generate_directory_structure=False, custom=OUR_PYFUNCEBLE_CONFIG)

## We can then play with PyFunceble and/or other business logic ...

Locally¶

To set globally simply parse your configuration along with the test method.

As example, you can do it like follow:

# We import the test method.
from PyFunceble import test as AvailabilityTest

# We set our list of indexes to overwritte.
OUR_PYFUNCEBLE_CONFIG = {"share_logs":False, "no_files": True}

# We get the status and parse our configuration.
status = AvailabilityTest("hello.world", config=OUR_PYFUNCEBLE_CONFIG)

## We can then manipulate the status and/or other business logic ...

Check the syntax of domains¶

Note

This example can be found in our examples repository.

"""
This is an example which respond to the following problematic(s):

    * How can I check the syntax of a domain with PyFunceble ?
"""

# We want some coloration so we import the tool do to that :)
from PyFunceble import initiate_colorama, Fore, Style
# We import the tool to print the colored CLI logo.
from PyFunceble.cli_core import CLICore
# We import the configuration loader.
from PyFunceble import load_config
# We import the test method of the PyFunceble API.
from PyFunceble import is_domain as PyFunceble

# We initiate the list of domains we are going to test.
DOMAINS = [
    "google.com",
    "tweeetttter.com",
    "github.com",
    "examplessss.ooooorgg",
    "twitter.com",
    "forest-jump",
]


# We initiate colorama.
initiate_colorama(True)

# We load our configuration.
#
# Note: We need this to print the logo but if you
# doesn't need the logo, you can ignore this.
load_config(generate_directory_structure=False)

# We print the PyFunceble logo.
CLICore.colorify_logo(home=True)

def print_syntax_result(subject, status):
    """
    Given the subject and its validation, we print it to STDOUT.

    :param str subject: The subject we are going to print.
    :param bool status: The validation state.
    """

    if status is True:
        print(f"{Fore.GREEN + Style.BRIGHT}{subject} is VALID")
    else:
        print(f"{Fore.CYAN + Style.BRIGHT}{subject} is INVALID")

for domain in DOMAINS:
    # We loop through the list of domain.

    # And we print the domain and status with the right coloration!
    print_syntax_result(domain, PyFunceble(domain))

Check the syntax of IPv4s¶

Note

This example can be found in our examples repository.

"""
This is an example which respond to the following problematic(s):

    * How can I check the syntax of an IPv4/IPv6 with PyFunceble ?
"""

# We want some coloration so we import the tool do to that :)
from PyFunceble import initiate_colorama, Fore, Style
# We import the tool to print the colored CLI logo.
from PyFunceble.cli_core import CLICore
# We import the configuration loader.
from PyFunceble import load_config
# We import the test method of the PyFunceble API.
from PyFunceble import is_ip as PyFunceble

# We initiate the list of IPs we are going to test.
IPS = ["216.58.207.46", "257.58.207.46"]


# We initiate colorama.
initiate_colorama(True)

# We load our configuration.
#
# Note: We need this to print the logo but if you
# doesn't need the logo, you can ignore this.
load_config(generate_directory_structure=False)

# We print the PyFunceble logo.
CLICore.colorify_logo(home=True)

def print_syntax_result(subject, status):
    """
    Given the subject and its validation, we print it to STDOUT.

    :param str subject: The subject we are going to print.
    :param bool status: The validation state.
    """

    if status is True:
        print(f"{Fore.GREEN + Style.BRIGHT}{subject} is VALID")
    else:
        print(f"{Fore.CYAN + Style.BRIGHT}{subject} is INVALID")

for ip in IPS:
    # We loop through the list of IP.

    # And we print the IP and status with the right coloration!
    print_syntax_result(ip, PyFunceble(ip))

Check the syntax of URLs¶

Note

This example can be found in our examples repository.

"""
This is an example which respond to the following problematic(s):

    * How can I check the syntax of an URL with PyFunceble ?
"""

# We want some coloration so we import the tool do to that :)
from PyFunceble import initiate_colorama, Fore, Style
# We import the tool to print the colored CLI logo.
from PyFunceble.cli_core import CLICore
# We import the configuration loader.
from PyFunceble import load_config
# We import the test method of the PyFunceble API.
from PyFunceble import is_url as PyFunceble

# We initiate the list of URLs we are going to test.
URLS = [
    "https://google.com",
    "http://tweeetttter.com",
    "htp://github.com",
    "httpp://examplessss.ooooorgg",
    "https:///twitter.com",
    "http:forest-jump",
]


# We initiate colorama.
initiate_colorama(True)

# We load our configuration.
#
# Note: We need this to print the logo but if you
# doesn't need the logo, you can ignore this.
load_config(generate_directory_structure=False)

# We print the PyFunceble logo.
CLICore.colorify_logo(home=True)

def print_syntax_result(subject, status):
    """
    Given the subject and its validation, we print it to STDOUT.

    :param str subject: The subject we are going to print.
    :param bool status: The validation state.
    """

    if status is True:
        print(f"{Fore.GREEN + Style.BRIGHT}{subject} is VALID")
    else:
        print(f"{Fore.CYAN + Style.BRIGHT}{subject} is INVALID")

for url in URLS:
    # We loop through the list of URL.

    # And we print the URL and status with the right coloration!
    print_syntax_result(url, PyFunceble(url))

Usage¶

From a terminal¶

--help¶

-v | --version¶

Source¶

-d "something" | --domain "something"¶

-url "something" | --url "something"¶

-f "something" | --file "something"¶

-uf "something" | --url-file "something"¶

Source filtering, decoding, conversion and expansion¶

--adblock¶

--complements¶

--filter "something"¶

--mining¶

--rpz¶

--wildcard¶

Test control¶

-c | --auto-continue | --continue¶

--cooldown-time¶

--local¶

--dns-lookup¶

--http-status-code-lookup | --http¶

--netinfo-lookup¶

--special-lookup¶

--whois-lookup¶

--reputation-lookup¶

--reputation¶

--syntax¶

-t "something" | --timeout "something"¶

-ua "something" | --user-agent "something"¶

-vsc | --verify-ssl-certificate¶

DNS control¶

--dns¶

--dns-protocol¶

Databases¶

--inactive-database¶

--database-type¶

-dbr "something" | --days-between-db-retest "something"¶

-wdb | --whois-database¶

Output control¶

-a | --all¶

-ex | --execution¶

--color | --colour¶

--display-status¶

--hierarchical¶

-h | --host¶

-ip "something" | –hosts-ip “something”¶

--no-files¶

--output-location¶

--unified-results¶

--percentage¶

--plain¶

--dots¶

-q | --quiet¶

--share-logs¶

-s | --simple¶

Multithreading¶

-w | --max-workers¶

CI / CD¶

--ci-max-minutes¶

--ci¶

--ci-branch¶

--ci-distribution-branch¶

--ci-command "something" | --cmd "something"¶

--ci-end-command "something" | --cmd-before-end "something"¶

--ci-commit-message "something" | --commit-autosave-message "something"¶

--ci-end-commit-message "something" | --commit-results-message "something"¶

Global overview¶

From a GitLab CI/CD environment¶

Configuration¶

Personal Access Token¶

.gitlab-ci.yml¶

From a Travis CI container¶

Configuration¶

Getting a GitHub token¶

Encrypting the token for future usage under the Travis CIs’ containers¶

Using the PyFunceble (Python) API¶

Get the availability of domains or IP¶

Get the availability of URL¶

Complete dataset while getting the avaibility of domains, IPs or URL¶

`--help`¶

`-v` | `--version`¶

`-d "something"` | `--domain "something"`¶

`-url "something"` | `--url "something"`¶

`-f "something"` | `--file "something"`¶

`-uf "something"` | `--url-file "something"`¶

`--adblock`¶

`--complements`¶

`--filter "something"`¶

`--mining`¶

`--rpz`¶

`--wildcard`¶

`-c` | `--auto-continue` | `--continue`¶

`--cooldown-time`¶

`--local`¶

`--dns-lookup`¶

`--http-status-code-lookup` | `--http`¶

`--netinfo-lookup`¶

`--special-lookup`¶

`--whois-lookup`¶

`--reputation-lookup`¶

`--reputation`¶

`--syntax`¶

`-t "something"` | `--timeout "something"`¶

`-ua "something"` | `--user-agent "something"`¶

`-vsc` | `--verify-ssl-certificate`¶

`--dns`¶

`--dns-protocol`¶

`--inactive-database`¶

`--database-type`¶

`-dbr "something"` | `--days-between-db-retest "something"`¶

`-wdb` | `--whois-database`¶

`-a` | `--all`¶

`-ex` | `--execution`¶

`--color` | `--colour`¶

`--display-status`¶

`--hierarchical`¶

`-h` | `--host`¶

`-ip "something"` | –hosts-ip “something”¶

`--no-files`¶

`--output-location`¶

`--unified-results`¶

`--percentage`¶

`--plain`¶

`--dots`¶

`-q` | `--quiet`¶

`--share-logs`¶

`-s` | `--simple`¶

`-w` | `--max-workers`¶

`--ci-max-minutes`¶

`--ci`¶

`--ci-branch`¶

`--ci-distribution-branch`¶

`--ci-command "something"` | `--cmd "something"`¶

`--ci-end-command "something"` | `--cmd-before-end "something"`¶

`--ci-commit-message "something"` | `--commit-autosave-message "something"`¶

`--ci-end-commit-message "something"` | `--commit-results-message "something"`¶

`.gitlab-ci.yml`¶