Usage

From a terminal

--help

Show the help message and exit.

-v | --version

Show the version of PyFunceble and exit.

Source

-d "something" | --domain "something"

Test one or more domains, separated by spaces.

Note

This argument takes 1 or more values.

As example:

$ PyFunceble -d example.org example.net

Note

When this option is used, no output files are generated.

-url "something" | --url "something"

Test one or more full URL, separated by spaces.

Note

When we test the availability of a URL, we (only) check the HTTP status code of the given URL.

Note

This argument takes 1 or more values.

As example:

$ PyFunceble -u https://example.org https://example.com

-f "something" | --file "something"

Read a local or remote (RAW link) file and test all domains inside it. If remote (RAW link) file is given, PyFunceble will download it, and test the content of the given RAW link as if it was a locally stored file.

Note

The system understands the following format:

  • plain list of subjects.
  • hosts file format.

-uf "something" | --url-file "something"

Read a local or remote (RAW link) file and test all (full) URLs inside it. If remote (RAW link) file is given, PyFunceble will download it, and test the content of the given RAW link as if it was a locally stored file.

Note

We consider one line as one URL to test.

Note

This argument test if a URL which is inside the given file is available. It ONLY tests full URLs.

As example:

$ PyFunceble -uf `https://raw.githubusercontent.com/funilrys/PyFunceble/dev/.travis/lists/url`

will download the given URL and test for its content assuming that each line represents a URL to test.

Warning

A test with this argument consists of the comparison of the status code. No WHOIS record will be requested nor DNS Lookup will be done.

Source filtering, decoding, conversion and expansion

-ad | --adblock

Switch the decoding of the adblock format.

Default value: False

If this argument is activated the system will extract all domains or IP from the given adblock file.

--complements

Switch the value of the generation and test of the complements. A complement is for example example.org if www.example.org is given and vice-versa.

Default value: False

--filter "something"

Domain to filter (regex).

Want to test all blogspot from your list? This argument allows you to do that!

Note

This argument should be a regex expression.

--idna

Switch the value of the IDNA conversion.

Default value: False

This argument allows the conversion of the domains using domain2idna

--mining

Switch the value of the mining subsystem usage.

Default value: False

Want to find domain or URL linked to a domain in your list? This argument will exactly do that.

Test control

-c | --auto-continue | --continue

Switch the value of the auto continue mode.

Default value: True

This argument activates or deactivates the auto-continue subsystem. Indeed, as we can automatically continue if the script has been stopped, this switch allows us to disable or enable the usage of that specific subsystem.

--cooldown-time

Switch the value of the cool-down time to apply between each test.

Default value: None

This argument applies a number of seconds to sleep before/between each test.

--http

Switch the value of the usage of HTTP code.

Default value: True

You don’t want to take the result of the HTTP code execution into consideration? This argument allows you to disable that!

Note

If activated the subsystem will bypass the HTTP status code extraction logic-representation.rst

--local

Switch the value of the local network testing.

Default value: False

Want to run a test over a local or private network? This argument will disable the limitation which does not apply to private networks.

-ns | --no-special

Switch the value of the usage of the SPECIAL rules.

Default value: False

Don’t want to use/apply the SPECIAL rules - which are explained in the source column section? This argument disables them all.

-nw | --no-whois

Switch the value of the usage of WHOIS to test the domain’s status.

Default value: False

Don’t want to use or take into consideration the results from whois? This argument allows you to disable it!

--reputation

Switch the value of the reputation test mode.

Default value: False

Warning

This will disable all other forms of test, will check against AlienVault’s reputation data and output its result into output/*/{MALICIOUS,SANE}/*.

--rpz

Switch the value of the RPZ policies test.

Default value: False

--shadow-file | --shadow

Switch the value of the usage and generation of a shadow file before a file test starts.

A shadow file is a file which only contain the actual list of subject to test. For its generation we check each subjects as we normally do on-the-fly.

Default value: False

Note

The shadow file, will just contain the actual list of subjects to test.

--syntax

Switch the value of the syntax test mode.

Default value: False

Warning

This will disable all other forms of test, will validate the syntax of a given test subject, and output its results in plain format into output/domains/{VALID,INVALID}/list

-t "something" | --timeout "something"

Switch the value of the timeout in seconds.

Default value: 5

This argument will set the default timeout to apply everywhere it is possible to set a timeout.

--use-reputation-data

Switch the value of the reputation data usage.

Default value: False

Warning

This only affects when used along with the availability test.

-ua "something" | --user-agent "something"

Set the user-agent to use and set every time we interact with everything which is not the logs sharing system.

-vsc | --verify-ssl-certificate

Switch the value of the verification of the SSL/TLS certificate when testing for URL.

Default value: False

Warning

If you activate the verification of the SSL/TLS certificate, you may get false-positive results.

Indeed if the certificate is not registered to the CA or is simply invalid and the domain is still alive, you will always get INACTIVE as output.

--wildcard

Switch the value of the wildcards test.

Default value: False

DNS (resolver) control

--dns

Set one or more specific DNS servers to use during the test. Separated by spaces.

Default value: Follow OS DNS ==> None

Warning

We expect a DNS server(s). If no DNS server(s) is given. You’ll almost for certain get all results as INACTIVE

This could happen in case you use --dns -f

Note

You can specify a port number to use to the DNS server if needed.

As example:

- 127.0.1.53:5353

--dns-lookup-over-tcp

Make all DNS queries through TCP instead of UDP.

Default value: False

Databases

-db | --database

Switch the value of the usage of a database to store inactive domains of the currently tested list.

Default value: True

This argument will disable or enable the usage of a database which saves all INACTIVE and INVALID domain of the given file over time.

Note

The database is retested every x day(s), where x is the number set in -dbr "something".

--database-type

Tell us the type of database to use. You can choose between the following: json, mariadb, mysql.

Default value: json

This argument let us use different types of database.

Note

This feature is applied to the following subsystems:

  • Autocontinue physically located (JSON) at output/continue.json.
  • InactiveDB physically located (JSON) at [config_dir]/inactive_db.json.
  • Mining physically located (JSON) at [config_dir]/mining.json.
  • WhoisDB physically located (JSON) at [config_dir]/whois.json.

-dbr "something" | --days-between-db-retest "something"

Set the numbers of days between each retest of domains present into the database of INACTIVE and INVALID domains.

Default value: 1

Note

This argument is only used if -db or inactive_database : true (under .PyFunceble.yaml) are activated.

-dbc "something" | --days-between-db-clean

Set the numbers of days since the introduction of a subject into inactive-db.json for it to qualifies for deletion.

Default value: 28

Note

This argument is only used if -db or inactive_database : true (under .PyFunceble.yaml) are activated.

-wdb | --whois-database

Switch the value of the usage of a database to store whois data to avoid whois servers rate limit.

Default value: True

Output control

-a | --all

Output all available information on the screen.

Default value: False

When activated:

Domain                        Status      Expiration Date   Source     HTTP Code
----------------------------- ----------- ----------------- ---------- ----------
pyfunceble.readthedocs.io     ACTIVE      Unknown           NSLOOKUP   302

When deactivated:

Domain                        Status      HTTP Code
----------------------------- ----------- ----------
pyfunceble.readthedocs.io     ACTIVE      302

--dots

Output dots (.) to stdout instead of giving the impression that we hang on.

Default value: False

-ex | --execution

Switch the default value of the execution time showing.

Default value: False

Want to know the execution time of your test? Well, this argument will let you know!

--hierarchical

Switch the value of the hierarchical sorting of the tested file.

Default value: True

This argument will output the result listed in a hierarchical order.

-h | --host

Switch the value of the generation of hosts file.

Default value: True

This argument will let the system know if it has to generate the hosts file version of each status.

-ip "something"

Change the IP to print in the hosts files with the given one.

Default value: 0.0.0.0

--json

Switch the value of the generation of the JSON formatted list of domains.

Default value: False

--less

When activated:

Domain                                                 Status      HTTP Code
------------------------------------------------------ ----------- ----------
pyfunceble.readthedocs.io                              ACTIVE      302

When deactivated:

Domain                       Status      Expiration Date   Source     HTTP Code
---------------------------- ----------- ----------------- ---------- ----------
pyfunceble.readthedocs.io    ACTIVE      Unknown           NSLOOKUP   302

-nf | --no-files

Switch the value of the production of output files.

Default value: False

Want to disable the production of the outputted files? This argument is for you!

-nl | --no-logs

Switch the value of the production of logs files in the case we encounter some errors.

Default value: False

Don’t want any logs to go out of PyFunceble? This argument disables every log subsystem.

-nu | --no-unified

Switch the value of the production unified logs under the output directory.

Default value: True

This argument disables the generation of result.txt.

--percentage

Switch the value of the percentage output mode.

Default value: True

This argument will disable or enable the generation of the percentage of each status.

--plain

Switch the value of the generation of the plain list of domains.

Default value: False:

Want to get a list with all domains for each status? The activation of this argument does the work while testing!

-q | --quiet

Run the script in quiet mode.

Default value: False

You prefer to run a program silently? This argument is for you!

--share-logs

Switch the value of the sharing of logs.

Default value: False

Want to make PyFunceble a better tool? Share your logs with our API which collect all logs!

-s | --simple

Switch the value of the simple output mode.

Default value: False

Want as less as possible data on screen? This argument returns as less as possible on screen!

--split

Switch the value of the split of the generated output.

Default value: True

Want to get the logs (copy of what you see on screen) on different files? This argument is suited to you!

--store-whois

Switch the value of the WHOIS record storage in the WHOIS DB.

Default value: False

The difference between False or True is whether we are saving a full dump of the WHOIS reply into the database.

If you for some reason believes you need to fill up your database with a complete dump of the whois reply, this is the right value to switch on.

Warning

Before switching this value, you should read these comments carefully…

You can test the amount of data by running whois mypdns.org from your Linux terminal, to see an example of what will be stored in the database.

You’re hearby warned…

store_whois_record comment

Brainstorm whois data comment

Multiprocessing

-m | --multiprocess

Switch the value of the usage of multiple processes.

Default value: False

Want to speed up the test time? This argument will allow the usage of multiple processes for testing.

--multiprocess-merging-mode

Sets the multiprocess merging mode. You can choose between the following live|ends.

Default value: end

Note

With the end value, the merging of cross-process data is made at the very end of the current instance.

Note

With the live value, the merging of cross-process data is made after the processing of the maximal number of processes.

Which means that if you allow 5 processes, we will run 5 tests, merge, run 5 tests, merge and so on until the end.

-p | --processes

Set the number of simultaneous processes to use while using multiple processes.

Default value: 25

Warning

DO not try to exceed your number of CPU if you want to keep your machine somehow alive and healthy!!

Note

If omitted, the number of available CPU cores will be used instead.

CI / CD

--autosave-minutes

Update the minimum of minutes before we start committing to upstream under the CI mode.

Default value: 15

--ci

Switch the value of the CI mode.

Default value: False

Note

If you combine this argument with the --quiet argument, the test will output a dotted line, where each dot (.) represent one test result or input which was skipped because it was previously tested.

Want to use PyFunceble under a supporter CI infrastructure/network? This argument is suited for your needs!

--ci-branch

Switch the branch name where we are going to push the temporary results.

Default value: master

Note

Currently the branch need to exist, but there are being worked on a path to have PyFunceble to create the sub-branch and finally merge it into the --ci-distribution-branch

--ci-distribution-branch

Switch the branch name where we are going to push the final results.

Default value: master

Note

The difference between this and --ci-branch is the fact that this branch will get the (final) result only when the test is finished under the given --ci-branch.

As an example, this allows us to have 2 branches:

  • proceessing (CI branch), for the tests with PyFunceble.
  • master (CI distribution branch), for the distribution of the results of PyFunceble.

--cmd “something”

Pass a command before each commit (except the final one).

Default value: ''

Note

In this example, something should be a script or a program which have to be executed when we reached the end of the given file.

Note

This argument is only used if --ci or ci: true (under .PyFunceble.yaml) are activated.

--cmd-before-end "something"

Pass a command before the results (final) commit under the CI mode.

Default value: ''

Note

In this example, something should be a script or a program which have to be executed when we reached the end of the given file.

Note

This argument is only used if --ci or ci: true (under .PyFunceble.yaml) are activated.

--commit-autosave-message "something"

Replace the default autosave commit message.

Default value: PyFunceble - AutoSave

This argument allows us to set a custom commit message which is going to be used as a commit message when saving.

Note

This argument is only used if --ci or ci: true (under .PyFunceble.yaml) are used.

Note

This argument is only used if we have to split the work into multiple processes because a list is too long or the timeout is reached.

Warning

Please avoid the usage of [ci skip] here.

--commit-results-message "something"

Replace the default results (final) commit message.

Default value: PyFunceble - Results

Note

This argument is only used if --ci or ci: true (under .PyFunceble.yaml) are used.

Note

This argument is only used if we reached the end of the list we are or have to test.

Unique actions

--clean

Clean all files under the output directory.

As it is sometimes needed to clean our output/ directory, this argument does the job automatically.

Warning

This argument delete everything which are .keep or .gitignore

--clean-all

Clean all files under the output directory along with all file generated by PyFunceble.

Warning

This deletes almost everything we generated without any warning.

Note

We don’t delete the whois database file/table because they are (almost) static data which are shared across launches in your environment.

Warning

If you plan to clean manually do not delete the whois database file or table as it will make your test finish under a much longer time as usual for you.

Warning

If you don’t combine this argument alongside with the --database-type argument or its configurable equivalent, this argument will only clean the JSON formatted databases.

--directory-structure

Generate the directory and files that are needed and which does not exist in the current directory.

Want to start without anything? This argument generates the output directory automatically for you!

Note

In case of a file or directory not found issue, it’s recommended to remove the dir_structure.json along with the output/ directory before using this argument.

Global overview

usage: PyFunceble [-d DOMAIN [DOMAIN ...]] [-u URL [URL ...]] [-f FILE]
                [-uf URL_FILE] [-ad] [--complements] [--filter FILTER]
                [--idna] [--mining] [-c] [--cooldown-time COOLDOWN_TIME]
                [--http] [--local] [-ns] [-nw] [--reputation] [--rpz]
                [--shadow-file] [--syntax] [-t TIMEOUT]
                [--use-reputation-data] [-ua USER_AGENT] [-vsc] [--wildcard]
                [--dns DNS [DNS ...]] [--dns-lookup-over-tcp] [-db]
                [--database-type DATABASE_TYPE]
                [-dbr DAYS_BETWEEN_DB_RETEST] [-dbc DAYS_BETWEEN_DB_CLEAN]
                [-wdb] [-a] [-ex] [--hierarchical] [-h] [-ip IP] [--json]
                [--less] [-nf] [-nl] [-nu] [--percentage] [--plain] [--dots]
                [-q] [--share-logs] [-s] [--split] [--store-whois] [-m]
                [--multiprocess-merging-mode MULTIPROCESS_MERGING_MODE]
                [-p PROCESSES] [--autosave-minutes AUTOSAVE_MINUTES] [--ci]
                [--ci-branch CI_BRANCH]
                [--ci-distribution-branch CI_DISTRIBUTION_BRANCH]
                [--cmd CMD] [--cmd-before-end CMD_BEFORE_END]
                [--commit-autosave-message COMMIT_AUTOSAVE_MESSAGE]
                [--commit-results-message COMMIT_RESULTS_MESSAGE] [--clean]
                [--clean-all] [--directory-structure] [--help] [-v]

PyFunceble - The tool to check the availability or syntax of domain, IP or URL.

optional arguments:
    --help                Show this help message and exit.
    -v, --version         Show the version of PyFunceble and exit.

Source:
    -d DOMAIN [DOMAIN ...], --domain DOMAIN [DOMAIN ...]
                            Test one or more domains, separated by spaces.

                            When this option is used, no output files are generated.
    -u URL [URL ...], --url URL [URL ...]
                            Test one or more full URL, separated by spaces.
    -f FILE, --file FILE  Read a local or remote (RAW link) file and test all domains inside it.
                            If remote (RAW link) file is given, PyFunceble will download it,
                            and test the content of the given RAW link as if it was a locally stored file.
    -uf URL_FILE, --url-file URL_FILE
                            Read a local or remote (RAW link) file and test all (full) URLs inside it.
                            If remote (RAW link) file is given, PyFunceble will download it,
                            and test the content of the given RAW link as if it was a locally stored file.

                            This argument test if an URL is available. It ONLY test full URLs.

Source filtering, decoding, conversion and expansion:
    -ad, --adblock        Switch the decoding of the adblock format.
                            Configured value: False
    --complements         Switch the value of the generation and test of the complements.
                            A complement is for example `example.org` if `www.example.org` is given and vice-versa.
                            Configured value: False
    --filter FILTER       Domain to filter (regex).
    --idna                Switch the value of the IDNA conversion.
                            Configured value: False
    --mining              Switch the value of the mining subsystem usage.
                            Configured value: False

Test control:
    -c, --auto-continue, --continue
                            Switch the value of the auto continue mode.
                            Configured value: True
    --cooldown-time COOLDOWN_TIME
                            Switch the value of the cooldown time to apply between each test.
                            Configured value: None
    --http                Switch the value of the usage of HTTP code.
                            Configured value: True
    --local               Switch the value of the local network testing.
                            Configured value: True
    -ns, --no-special     Switch the value of the usage of the SPECIAL rules.
                            Configured value: False
    -nw, --no-whois       Switch the value of the usage of WHOIS to test the domain's status.
                            Configured value: False
    --reputation          Switch the value of the reputation test mode.
                            Configured value: False
    --rpz                 Switch the value of the RPZ policies test.

                            When used, RPZ policies will be properly tested.
    --shadow-file, --shadow
                            Switch the value of the usage and generation of a shadow file before a file test starts.

                            A shadow file is a file which only contain the actual list of subject to test. For its generation we check each subjects as we normally do on-the-fly.
                            Configured value: False
    --syntax              Switch the value of the syntax test mode.
                            Configured value: False
    -t TIMEOUT, --timeout TIMEOUT
                            Switch the value of the timeout in seconds.
                            Configured value: 5
    --use-reputation-data
                            Switch the value of the reputation data usage.
                            Configured value: False
    -ua USER_AGENT, --user-agent USER_AGENT
                            Set the user-agent to use and set every time we interact with everything which
                            is not the logs sharing system.
    -vsc, --verify-ssl-certificate
                            Switch the value of the verification of the SSL/TLS certificate when testing for URL.
                            Configured value: False
    --wildcard            Switch the value of the wildcards test.

                            When used, wildcards will be properly tested.

DNS (resolver) control:
    --dns DNS [DNS ...]   Set one or more DNS server(s) to use during testing. Separated by spaces.

                            To specify a port number for the DNS server you append it as :port [ip:port].

                            If no port is specified, the default DNS port (53) is used.
                            Configured value: OS (declared) DNS server
    --dns-lookup-over-tcp
                            Make all DNS queries with TCP.
                            Configured value: False

Databases:
    -db, --database       Switch the value of the usage of a database to store inactive domains of the currently tested list.
                            Configured value: True
    --database-type DATABASE_TYPE
                            Tell us the type of database to use.
                            You can choose between the following: `json | mariadb | mysql`
                            Configured value: 'json'
    -dbr DAYS_BETWEEN_DB_RETEST, --days-between-db-retest DAYS_BETWEEN_DB_RETEST
                            Set the numbers of days between each retest of domains present into inactive-db.json.
                            Configured value: 1
    -dbc DAYS_BETWEEN_DB_CLEAN, --days-between-db-clean DAYS_BETWEEN_DB_CLEAN
                            Set the numbers of days since the introduction of a subject into inactive-db.json for it to qualifies for deletion.
                            Configured value: 28
    -wdb, --whois-database
                            Switch the value of the usage of a database to store whois data to avoid whois servers rate limit.
                            Configured value: True

Output control:
    -a, --all             Output all available information on the screen.
                            Configured value: True
    -ex, --execution      Switch the default value of the execution time showing.
                            Configured value: False
    --hierarchical        Switch the value of the hierarchical sorting of the tested file.
                            Configured value: False
    -h, --host            Switch the value of the generation of hosts file.
                            Configured value: True
    -ip IP                Change the IP to print in the hosts files with the given one.
                            Configured value: '0.0.0.0'
    --json                Switch the value of the generation of the JSON formatted list of domains.
                            Configured value: False
    --less                Output less informations on screen.
                            Configured value: False
    -nf, --no-files       Switch the value of the production of output files.
                            Configured value: False
    -nl, --no-logs        Switch the value of the production of logs files in the case we encounter some errors.
                            Configured value: False
    -nu, --no-unified     Switch the value of the production unified logs under the output directory.
                            Configured value: False
    --percentage          Switch the value of the percentage output mode.
                            Configured value: True
    --plain               Switch the value of the generation of the plain list of domains.
                            Configured value: False
    --dots                Prints dots to stdout instead of giving the impression that we hang on.
                            Configured value: False
    -q, --quiet           Run the script in quiet mode.
                            Configured value: False
    --share-logs          Switch the value of the sharing of logs.
                            Configured value: False
    -s, --simple          Switch the value of the simple output mode.
                            Configured value: False
    --split               Switch the value of the split of the generated output files.
                            Configured value: True
    --store-whois         Switch the value of the WHOIS record storage in the WHOIS DB.
                            Configured value: False

Multiprocessing:
    -m, --multiprocess    Switch the value of the usage of multiple processes.
                            Configured value: False
    --multiprocess-merging-mode MULTIPROCESS_MERGING_MODE
                            Sets the multiprocess merging mode.
                            You can choose between the following: `live|ends`.
                            Configured value: 'end'
    -p PROCESSES, --processes PROCESSES
                            Set the number of simultaneous processes to use while using multiple processes.
                            If omited, the number of available CPU cores will be used instead.
                            Configured value: 25

CI / CD:
    --autosave-minutes AUTOSAVE_MINUTES
                            Update the minimum of minutes before we start committing to upstream under the CI mode.
                            Configured value: 15
    --ci                  Switch the value of the CI mode.
                            Configured value: False
    --ci-branch CI_BRANCH
                            Switch the branch name where we are going to push the temporary results.
                            Configured value: 'master'
    --ci-distribution-branch CI_DISTRIBUTION_BRANCH
                            Switch the branch name where we are going to push the final results.
                            Configured value: 'master'
    --cmd CMD             Pass a command to run before each commit (except the final one) under the CI mode.
                            Configured value: ''
    --cmd-before-end CMD_BEFORE_END
                            Pass a command to run before the results (final) commit under the CI mode.
                            Configured value: ''
    --commit-autosave-message COMMIT_AUTOSAVE_MESSAGE
                            Replace the default autosave commit message.
                            Configured value: None
    --commit-results-message COMMIT_RESULTS_MESSAGE
                            Replace the default results (final) commit message.
                            Configured value: None

Unique actions:
    --clean               Clean all files under the output directory.
    --clean-all           Clean all files under the output directory along with all file generated by PyFunceble.
    --directory-structure
                            Generate the directory and files that are needed and which does not exist in the current directory.

For an in-depth usage, explanation and examples of the arguments, you should read the documentation at https://pyfunceble.readthedocs.io/en/dev/

Crafted with ♥ by Nissar Chababy (@funilrys) with the help of https://pyfunceble.github.io/contributors.html && https://pyfunceble.github.io/special-thanks.html

From a GitLab CI/CD environment

As we offer an argument named --ci which will autosave in a GitLab CI/CI environment, this document try to describe hot it works!

Configuration

Personal Access Token

A personal access token is needed in order for PyFunceble to automatically push the results.

You should get a personal GitLab access token with the read_repository and write_repository scopes.

Once created and copied in a safe place, create a new masked variable named GL_TOKEN inside the CI/CD settings of your project. The value of the variable should be the newly generated personal access token.

.gitlab-ci.yml

Note

This part only present a commented .gitlab-ci.yml. This is just an example do not take the following as necessarly true.

You’re invited to submit changes if something stated in this document is wrong.

# Python needed, so we use the python image.
image: python:latest

variables:
    # This is the Git name we have to set. (git config user.name)
    GIT_EMAIL: "dead-hosts@funilrys.com"
    # This is the Git Email we have to set. (git config user.email)
    GIT_NAME: "GitLab CI/CD"

before_script:
    # We install the development version of PyFunceble.
    # If you prefer the stable version replace `pyfunceble-dev`
    # with `pyfunceble`.
    - pip3 install PyFunceble-dev

run:
    script:
        # Let's say we want our results and our PyFunceble
        # infrastructure to be saved in a directory called `PyFunceble-tests`

        # We move inside it.
        - cd PyFunceble-tests
        # We test the file `my_awesome_list` which is located inside the current directory.
        # Note: we precise the `--ci` argument here,
        #     but you work without it if you set `ci: true` inside your `.PyFunceble.yaml`
        - PyFunceble --ci -f my_awesome_list --plain

From a Travis CI container

As we offer an argument named --ci to activate the usage of PyFunceble in a Travis CI instance, we document here what you need to know!

Configuration

Note

This part only present a commented .travis.yml so that you can understand where to start.

If you need more practical examples, feel free to report to one of Dead-Hosts repositories which use PyFunceble with Travis CI.

env:
    global:
        # The following is your encrypted GitHub API key.
        # Indeed as we are going to push to the repository, this is needed.
        #- GH_TOKEN: # This can be set in the travis-ci https://travis-ci.com/repo/settings as 'Environment Variables'
        # or as below: secure: encrypted code
        - secure: QQdKFquFFojFT9XJ1XZp4EMoDTVoXFgqZq8XU+sCVf+pJQR6d/oKBp8rnSTCnZizWOQXUjGXUUxUpSG/dYGyBLjo3rH3rsn9ciZHVfubxbwK860w4sqibl4DvhCv2rdsFtvzXnhm4P9OL3i+krKdewh9fxpNyUU58qOgfnS7mK9FcFhb8z5ak2sxU2XRZedwm6Ro0oyVKs8kFkL4YaADfNyAHlGTfr9rVmE52WXQXQENktb9gFgR2A8ZnmLy0BCMZGkPDShJnjRDWD4DErtasLmLQvWpzOBwdbVJTY6U9KDRXVNdC9lp5E5Ba/dc0y36q6vjfgJR+QchetOtHgNbKYbLB8c26Di90OZCFJsxMNcl1Wct4qFPXkFGvjXrISW6pbdPL5Plto0Ig3iLiulhYOPVArysMIk9ymtSXP+WE7VWX01LQ1fEkIoSfeVZ2caTnCmTsoHVGRRe978CojKaT7yU45kb15hcyDrzptQ8EP2hfxeh5F7KtueQ6Rsb9LFDZMkMDKflZn6a+bRhESlmWWmYB9stzGzTurQA1E1bcSACJ8A8hG5nHBzZYJ2S+OY0PE7UdyOJ0JK0qe/67d+F9ocQdIoFpDDTdgIjHerQnD2wRg1aKPzLDb4jJTpqgr5ssPrqUAKl3st7gyaAZzCEADPDnIBDjOJS+mFWbx9DKgc=
        # This is the Git name we have to set. (git config user.name)
        - GIT_NAME: Travis CI
        # This is the Git Email we have to set. (git config user.email)
        - GIT_EMAIL: dead-hosts@funilrys.com

# This is the language we use.
language: python

# This is the python version we are going to use for the tests.
# Note: you can add any 3.x version to the list.
python:
- "3.8"

# The following will tell Travis CI to ends as fast as possible.
matrix:
    fast_finish: true

# Here we are setting what Travis CI have to cache.
cache:
    # We are caching pip3 as we use it to install PyFunceble
    - pip3

install:
    # We install the development version of PyFunceble. If you prefer the stable version replace
    # `pyfunceble-dev` with `pyfunceble`.
    - pip3 install pyfunceble-dev

# Our tests start here.
script:
    # Let's say we want our results and our PyFunceble infrastructure to be saved in a directory
    # called `PyFunceble-tests`

    # We move inside it.
    - cd PyFunceble-tests
    # We test the file `my_awesome_list` which is located inside the current directory.
    # Note: we precise the `--ci` argument here,
    #     but you work without it if you set `travis: true` inside your `.PyFunceble.yaml`
    - PyFunceble --ci -f my_awesome_list --plain

# The following initiate email notification logic.
notifications:
    # As we want to get a mail on failure and on status change, we set the following.
    on_success:   change
    on_failure:   always

Getting a GitHub token

For the secure index of the .travis.yml file, you have to generate a new GitHub token.

After you got your token, please write it or save it in a safe place as you’re going to need it every time you’re going to interact with Travis CI.

Note

The scope to set is public_repo but you can also set others depending on your needs.

Encrypting the token for future usage under the Travis CIs’ containers

To encrypt the token simply replace and execute the following according to your personal case.

$ travis encrypt 'GH_TOKEN=theGeneratedToken' -r 'The content of TRAVIS_REPO_SLUG' --add

Warning

Please do not execute the following explicitly without replacing theGeneratedToken with your previously generated GitHub token and The content of TRAVIS_REPO_SLUG with your repository slug.

Note

The usage of --add ensure that the travis program automatically add the secure index to the .travis.yml file.

Using the PyFunceble (Python) API

If you are working with a python script, module or even class, you can integrate PyFunceble to your main logic by importing it and using its API (cf: API Documentation).

This section will present some example of the way you can interact with PyFunceble from anything written in Python.

Get the availability of domains or IP

Note

This example can be found in our examples repository.

Todo

Add IPs in the loop.

"""
This is an example which respond to the following problematic(s):

    * How can I get the avaibility of a domain or IP with PyFunceble ?
"""

# We want some coloration so we import the tool do to that :)
from PyFunceble import initiate_colorama, Fore, Style
# We import the tool to print the colored CLI logo.
from PyFunceble.cli_core import CLICore
# We import the configuration loader.
from PyFunceble import load_config
# We import the test method of the PyFunceble API.
from PyFunceble import test as PyFunceble

# We initiate the list of domains we are going to test.
DOMAINS = [
    "google.com",
    "tweeetttter.com",
    "github.com",
    "examplessss.ooooorgg",
    "twitter.com",
    "forest-jump"
]

# We initiate colorama.
initiate_colorama(True)

# We load our configuration.
#
# Note: We need this to print the logo but if you
# doesn't need the logo, you can ignore this.
load_config(generate_directory_structure=False)

# We print the PyFunceble logo.
CLICore.colorify_logo(home=True)

def print_result(subject, status):
    """
    Given the subject and its status, we print it to STDOUT.

    :param str subject: The subject we are going to print.
    :param str status: The status of the domain.
    """

    if status == "ACTIVE":
        print(f"{Fore.GREEN + Style.BRIGHT}{domain} is {status}")
    elif status == "INACTIVE":
        print(f"{Fore.RED + Style.BRIGHT}{domain} is {status}")
    else:
        print(f"{Fore.CYAN + Style.BRIGHT}{domain} is {status}")

for domain in DOMAINS:
    # We loop through the list of domain.

    # And we print the domain and status with the right coloration!
    print_result(domain, PyFunceble(domain))

Get the availability of URL

Note

This example can be found in our examples repository.

"""
This is an example which respond to the following problematic(s):

    * How can I get the avaibility of an URL with PyFunceble ?
"""

# We want some coloration so we import the tool do to that :)
from PyFunceble import initiate_colorama, Fore, Style
# We import the tool to print the colored CLI logo.
from PyFunceble.cli_core import CLICore
# We import the configuration loader.
from PyFunceble import load_config
# We import the test method of the PyFunceble API.
from PyFunceble import url_test as PyFunceble

# We initiate the list of URLs we are going to test.
URLS = [
    "https://google.com",
    "http://tweeetttter.com",
    "ftp://github.com",
    "http://examplessss.ooooorgg",
    "http://twitter.com",
]

# We initiate colorama.
initiate_colorama(True)

# We load our configuration.
#
# Note: We need this to print the logo but if you
# doesn't need the logo, you can ignore this.
load_config(generate_directory_structure=False)

# We print the PyFunceble logo.
CLICore.colorify_logo(home=True)

def print_result(subject, status):
    """
    Given the subject and its status, we print it to STDOUT.

    :param str subject: The subject we are going to print.
    :param str status: The status of the domain.
    """

    if status == "ACTIVE":
        print(f"{Fore.GREEN + Style.BRIGHT}{domain} is {status}")
    elif status == "INACTIVE":
        print(f"{Fore.RED + Style.BRIGHT}{domain} is {status}")
    else:
        print(f"{Fore.CYAN + Style.BRIGHT}{domain} is {status}")

for url in URLS:
    # We loop through the list of domain.

    # And we print the domain and status with the right coloration!
    print_result(url, PyFunceble(url))

Complete dataset while getting the avaibility of domains, IPs or URL

While using our API, you can request to see/get everything with the help of the complete=True argument.

You’ll then get the following dict as output.

{
    "_status": None, # If some extra rules are applied, this index will keep the status before the extra rules was applied.
    "_status_source": None, # If some extra rules are applied, this index will keep the source before the extra rules was applied.
    "domain_syntax_validation": None, # The domain syntax validation status.
    "expiration_date": None, # The expiration date of the tested subject (if found).
    "http_status_code": None, # The status code of the tested subejct.
    "ip4_syntax_validation": None, # The IPv4 syntax validation status.
    "dns_lookup": [], # The DNS Lookup output.
    "status_source": None, # The (final) source which gave us the status.
    "status": None, # The (final) status returned by PyFunceble.
    "tested": None, # The tested subject.
    "url_syntax_validation": None, # The url syntax validation status.
    "whois_record": None, # The whois record (if found).
    "whois_server": None, # The whois server we use to get the whois record (if found).
}

Set custom configuration index while getting the avaibility of domains, IPs or URL

While using PyFunceble, you might want to set or overwritte a default behaviour.

You can do that in 2 ways. Globally or locally.

Globally

To set globally simply initiate the configuration loader and parse your custom configuration along with the initialization.

As example, you can do it like follow:

# We import the configuration loader.
from PyFunceble import load_config

# We set our list of indexes to overwritte.
OUR_PYFUNCEBLE_CONFIG = {"share_logs":False, "no_files": True}

# We load our configuration and parse our custom indexes.
load_config(generate_directory_structure=False, custom=OUR_PYFUNCEBLE_CONFIG)

## We can then play with PyFunceble and/or other business logic ...

Locally

To set globally simply parse your configuration along with the test method.

As example, you can do it like follow:

# We import the test method.
from PyFunceble import test as AvailabilityTest

# We set our list of indexes to overwritte.
OUR_PYFUNCEBLE_CONFIG = {"share_logs":False, "no_files": True}

# We get the status and parse our configuration.
status = AvailabilityTest("hello.world", config=OUR_PYFUNCEBLE_CONFIG)

## We can then manipulate the status and/or other business logic ...

Check the syntax of domains

Note

This example can be found in our examples repository.

"""
This is an example which respond to the following problematic(s):

    * How can I check the syntax of a domain with PyFunceble ?
"""

# We want some coloration so we import the tool do to that :)
from PyFunceble import initiate_colorama, Fore, Style
# We import the tool to print the colored CLI logo.
from PyFunceble.cli_core import CLICore
# We import the configuration loader.
from PyFunceble import load_config
# We import the test method of the PyFunceble API.
from PyFunceble import is_domain as PyFunceble

# We initiate the list of domains we are going to test.
DOMAINS = [
    "google.com",
    "tweeetttter.com",
    "github.com",
    "examplessss.ooooorgg",
    "twitter.com",
    "forest-jump",
]


# We initiate colorama.
initiate_colorama(True)

# We load our configuration.
#
# Note: We need this to print the logo but if you
# doesn't need the logo, you can ignore this.
load_config(generate_directory_structure=False)

# We print the PyFunceble logo.
CLICore.colorify_logo(home=True)

def print_syntax_result(subject, status):
    """
    Given the subject and its validation, we print it to STDOUT.

    :param str subject: The subject we are going to print.
    :param bool status: The validation state.
    """

    if status is True:
        print(f"{Fore.GREEN + Style.BRIGHT}{subject} is VALID")
    else:
        print(f"{Fore.CYAN + Style.BRIGHT}{subject} is INVALID")

for domain in DOMAINS:
    # We loop through the list of domain.

    # And we print the domain and status with the right coloration!
    print_syntax_result(domain, PyFunceble(domain))

Check the syntax of IPv4s

Note

This example can be found in our examples repository.

"""
This is an example which respond to the following problematic(s):

    * How can I check the syntax of an IPv4/IPv6 with PyFunceble ?
"""

# We want some coloration so we import the tool do to that :)
from PyFunceble import initiate_colorama, Fore, Style
# We import the tool to print the colored CLI logo.
from PyFunceble.cli_core import CLICore
# We import the configuration loader.
from PyFunceble import load_config
# We import the test method of the PyFunceble API.
from PyFunceble import is_ip as PyFunceble

# We initiate the list of IPs we are going to test.
IPS = ["216.58.207.46", "257.58.207.46"]


# We initiate colorama.
initiate_colorama(True)

# We load our configuration.
#
# Note: We need this to print the logo but if you
# doesn't need the logo, you can ignore this.
load_config(generate_directory_structure=False)

# We print the PyFunceble logo.
CLICore.colorify_logo(home=True)

def print_syntax_result(subject, status):
    """
    Given the subject and its validation, we print it to STDOUT.

    :param str subject: The subject we are going to print.
    :param bool status: The validation state.
    """

    if status is True:
        print(f"{Fore.GREEN + Style.BRIGHT}{subject} is VALID")
    else:
        print(f"{Fore.CYAN + Style.BRIGHT}{subject} is INVALID")

for ip in IPS:
    # We loop through the list of IP.

    # And we print the IP and status with the right coloration!
    print_syntax_result(ip, PyFunceble(ip))

Check the syntax of URLs

Note

This example can be found in our examples repository.

"""
This is an example which respond to the following problematic(s):

    * How can I check the syntax of an URL with PyFunceble ?
"""

# We want some coloration so we import the tool do to that :)
from PyFunceble import initiate_colorama, Fore, Style
# We import the tool to print the colored CLI logo.
from PyFunceble.cli_core import CLICore
# We import the configuration loader.
from PyFunceble import load_config
# We import the test method of the PyFunceble API.
from PyFunceble import is_url as PyFunceble

# We initiate the list of URLs we are going to test.
URLS = [
    "https://google.com",
    "http://tweeetttter.com",
    "htp://github.com",
    "httpp://examplessss.ooooorgg",
    "https:///twitter.com",
    "http:forest-jump",
]


# We initiate colorama.
initiate_colorama(True)

# We load our configuration.
#
# Note: We need this to print the logo but if you
# doesn't need the logo, you can ignore this.
load_config(generate_directory_structure=False)

# We print the PyFunceble logo.
CLICore.colorify_logo(home=True)

def print_syntax_result(subject, status):
    """
    Given the subject and its validation, we print it to STDOUT.

    :param str subject: The subject we are going to print.
    :param bool status: The validation state.
    """

    if status is True:
        print(f"{Fore.GREEN + Style.BRIGHT}{subject} is VALID")
    else:
        print(f"{Fore.CYAN + Style.BRIGHT}{subject} is INVALID")

for url in URLS:
    # We loop through the list of URL.

    # And we print the URL and status with the right coloration!
    print_syntax_result(url, PyFunceble(url))