The tool to check the availability or syntax of domains, IPv4 or URL.¶

https://az743702.vo.msecnd.net/cdn/kofi3.png

https://api.travis-ci.com/funilrys/PyFunceble.png?branch=master

https://coveralls.io/repos/github/funilrys/PyFunceble/badge.png?branch=master

https://img.shields.io/github/license/funilrys/PyFunceble.png

https://img.shields.io/github/release/funilrys/PyFunceble/all.png

https://img.shields.io/github/issues/funilrys/PyFunceble.png

https://img.shields.io/badge/code%20style-black-000000.png

PyFunceble is the little sister of Funceble which was archived on 13th March 2018.

Its main objective is to provide the availability of domains, IPs and since recently URL by generating an accurate result based on results from WHOIS, NSLOOKUP and HTTP status codes.

PyFunceble is currently running actively and daily with the help of Travis CI under 60+ repositories. It is used to clean or test the availability of data which are present in hosts files, list of IP, list of domains, block lists or even AdBlock filter lists.

PyFunceble provides some useful features for continuous testing.

As an example, its auto-continue system coupled with its auto-save system allows it to run nice and smoothly under Travis CI without even reaching Travis CI time restriction. In the other side, its internal inactive database system let INACTIVE and INVALID caught domains, IPs or URLs being automatically retested over time on next run.

What can PyFunceble do?¶

Test of the availability of a domain.
Test of the availability of an IPv4.
Test of the availability of an URL.
Test of the availability of a domain/DNS name in a private or local network.
Test of the availability of an IPv4 in a private or local network.
Test of the availability of an URL in a private or local network.
Test of the syntax of a domain.
Test of the syntax of an IPv4.
Test of the syntax of an URL.
Test of domain or IP which are present into an Adblock formatted file.
Test from a given raw link.
Save test result(s) on file(s) (hosts file, plain text and/or JSON format).
Show test result(s) on screen.
Show percentage of each status (ACTIVE, INACTIVE, INVALID)
Sort outputs hierarchically.
“Mining” of domain or IP which are related to the tested element.
Auto-continuation of tests in case of system crash or script stop.
Filtering of a file content.
- This feature will let us for example test all blogspot domain of the given file no matter the content of the file.
Set the user-agent to use for the tests.
Give some analytic depending of the HTTP status code (ACTIVE, POTENTIALLY_ACTIVE, POTENTIALLY_INACTIVE, SUSPICIOUS).
Retest overtime of INACTIVE and INVALID domains.
Print the execution time on screen and file.
Customization of the different option via command-line arguments or configuration file.
Continuous tests under Travis CI with the help of an auto saving and database system.
- Set branch to push the result to for the autosaving system.
- Set the minimal time before we autosave in order to avoid Travis CI limitation.
- Set a command to execute at the end of the test.
- Set the commit message for the autosaving system.
… and a lot more!

PyFunceble in action¶

Domain testing¶

https://github.com/PyFunceble/gifs/raw/master/domain.gif

URL testing¶

https://github.com/PyFunceble/gifs/raw/master/url.gif

Domain and URL testing with simple output¶

https://github.com/PyFunceble/gifs/raw/master/simple.gif

File content testing¶

https://github.com/PyFunceble/gifs/raw/master/file.gif

File content testing with IDNA/Punycode conversion¶

https://github.com/PyFunceble/gifs/raw/master/file_idna.gif

Generated files/outputs¶

https://github.com/PyFunceble/gifs/raw/master/outputs.gif

Dead-Hosts as place to use PyFunceble!¶

Why Dead-Hosts?¶

We are conscient that PyFunceble may run for days, that’s why we offer you to request your list to be tested at Dead-Hosts.

How do Dead-Hosts work?¶

Dead-Hosts is like a wrapper of PyFunceble. Indeed, we use Travis CI containers in order to test lists with PyFunceble.

Once a list is set up inside our infrastructure, PyFunceble will test the list regularly and the Dead-Hosts infrastructure will produce a clean.list file which represents the list of domains/IP/URL which remains or became ACTIVE.

In addition, if needed, we can set up a list of GitHub username to @ping once a new test is finished.

History of Dead-Hosts¶

The project started on 3rd March 2017 at funilrys/dead-hosts.

The original idea was to test Funceble against hosts file in order to find bugs inside Funceble but also letting me have a fewer long hosts file.

On 23rd January 2018, I (funilrys) decided to shut funilrys/dead-hosts down. Indeed, as it became impossible to test all members of the project without having to wait weeks, I decided that it was time to move to another level.

That was the beginning of Dead-Hosts.

Today Dead-Hosts’s objective is to provide to project/list maintainers or individuals - with the help of PyFunceble - more information about their favorite project/list or domains, IP or URL.

Installation¶

Requirements¶

Here is the list of requirements:

Python 3.6+
colorama
domain2idna
PyYAML
requests
setuptools
urllib3

Python 3.6+¶

As we use for example

print('hello', end=' ')

which does not exist in Python 2.x and as I wanted to give a priority to Python 3, Python 3 is required.

`colorama`¶

As we use some coloration coloration, colorama is required.

`domain2idna`¶

As we propose the conversion of domains to IDNA, domain2idna is required.

Note

domain2idna is maintained and developed by Nissar Chababy (@funilrys), the main developer of PyFunceble. Its source code can be found on GitHub.

`PyYAML`¶

As our configuration file is written in .yaml, PyYAML is required.

`requests`¶

As we use requests when calling all Lookup() methods, requests is required.

`setuptools`¶

As we use install_requires=xx inside our setup.py, setuptools is required.

`urllib3`¶

You should normally already have it. But as we handle some of its errors while using requests, urllib3 is required.

Get PyFunceble¶

Stable version¶

Using `pip`¶

Choose your repository, install and enjoy PyFunceble!

From PyPi¶

$ pip3 install PyFunceble

From GitHub¶

$ pip3 install git+https://github.com/funilrys/PyFunceble.git@master#egg=PyFunceble

Using the AUR (for Arch Linux users)¶

The package can be found at https://aur.archlinux.org/packages/python-pyfunceble/.

With makepkg¶

$ wget https://aur.archlinux.org/cgit/aur.git/plain/PKGBUILD?h=python-pyfunceble
$ makepkg
$ sudo pacman -U python-pyfunceble*.tar.xz

With your favorite AUR helper¶

Warning

We do not recommend any AUR helper but keep in mind that some AUR helpers are “better” than other. For more information about your current (or any other) AUR helper please report to the ArchWiki page.

$ yourFavoriteAurHelper -S python-pyfunceble

Pure Python method¶

Execute the following and enjoy PyFunceble!

$ git clone https://github.com/funilrys/PyFunceble.git
$ cd PyFunceble
$ python3 setup.py test && python3 setup.py install

Development version¶

The development version of PyFunceble represents the dev branch. It’s intended for the development of next features but is always at a usable state.

Indeed, We should not push to the dev branch until we are sure that the new commit does not break or introduce critical issue under PyFunceble.

For development¶

Execute the following and let’s hack PyFunceble!

Note

We highly recommend you to develop PyFunceble under a virtualenv.

$ git clone https://github.com/funilrys/PyFunceble.git
$ cd PyFunceble && git checkout dev && virtualenv venv
$ source venv/bin/activate && pip3 install -e .

Note

After installing with:

$ source venv/bin/activate && pip3 install -e .

you only need to update the repository.
you don’t have to rerun the pip command.

For usage¶

Using `pip`¶

Execute one of the following and enjoy PyFunceble!

From PyPi

$ pip3 install PyFunceble-dev

From GitHub

$ pip3 install git+https://github.com/funilrys/PyFunceble.git@dev#egg=PyFunceble

Using the AUR (for Arch Linux users)¶

The package can be found at https://aur.archlinux.org/packages/python-pyfunceble-dev/.

With makepkg

$ wget https://aur.archlinux.org/cgit/aur.git/plain/PKGBUILD?h=python-pyfunceble-dev
$ makepkg
$ sudo pacman -U python-pyfunceble-dev*.tar.xz

With your favorite AUR helper

Warning

We do not recommend any AUR helper but keep in mind that some AUR helpers are “better” than other. For more information about your current (or any other) AUR helper please report to the ArchWiki page.

$ yourFavoriteAurHelper -S python-pyfunceble-dev

Pure Python method¶

Execute the following and enjoy PyFunceble!

$ git clone https://github.com/funilrys/PyFunceble.git
$ cd PyFunceble && git checkout dev
$ python3 setup.py test && python3 setup.py install

First steps¶

Make sure that you can run

$ PyFunceble --version

and enjoy PyFunceble!!

Update¶

Stable version¶

Using `pip`¶

Choose your repository, install and enjoy PyFunceble!

From PyPi¶

$ pip3 install --upgrade PyFunceble

From GitHub¶

$ pip3 install --upgrade git+https://github.com/funilrys/PyFunceble.git@master#egg=PyFunceble

Using the AUR (for Arch Linux users)¶

With makepkg¶

$ wget https://aur.archlinux.org/cgit/aur.git/plain/PKGBUILD?h=python-pyfunceble
$ makepkg
$ sudo pacman -U python-pyfunceble*.tar.xz

With your favorite AUR helper¶

Warning

We do not recommend any AUR helper but keep in mind that some AUR helpers are “better” than other. For more information about your current (or any other) AUR helper please report to the ArchWiki page.

$ yourFavoriteAurHelper -Syu python-pyfunceble

Pure Python method¶

Execute the following and enjoy PyFunceble!

$ cd PyFunceble
$ git checkout master && git fetch origin && git merge origin/master
$ python3 setup.py test && python3 setup.py install

Development version¶

For development¶

$ cd PyFunceble && git checkout dev
$ git fetch origin && git merge origin/dev

Note

As you previously installed with

$ . venv/bin/activate && pip3 install -e .

Only code/repository update is required.

For usage¶

Using `pip`¶

Execute one of the following and enjoy PyFunceble!

From PyPi

$ pip3 install --upgrade PyFunceble-dev

From GitHub

$ pip3 install --upgrade git+https://github.com/funilrys/PyFunceble.git@dev#egg=PyFunceble

Using the AUR (for Arch Linux users)¶

With makepkg

$ wget https://aur.archlinux.org/cgit/aur.git/plain/PKGBUILD?h=python-pyfunceble-dev
$ makepkg
$ sudo pacman -U python-pyfunceble-dev*.tar.xz

With your favorite AUR helper

Warning

We do not recommend any AUR helper but keep in mind that some AUR helpers are “better” than other. For more information about your current (or any other) AUR helper please report to the ArchWiki page.

$ yourFavoriteAurHelper -Syu python-pyfunceble-dev

Pure Python method¶

Execute the following and enjoy PyFunceble!

$ cd PyFunceble && git checkout dev
$ git fetch origin && git merge origin/dev
$ python3 setup.py test && python3 setup.py install

Configuration¶

This page will try to detail each configuration available into .PyFunceble.yaml along with the location of where we are looking for the configuration file.

Location¶

Problematics¶

How can we create a more efficient way to work with configuration?
How can we make the configuration file(s) available globally so that PyFunceble can be run everywhere in the user workspace?

To answer those problematics, we moved the configuration location elsewhere in the place where most users expect to have their configuration file(s).

Clone¶

If you cloned the repository and you’re trying to test from a cloned directory (the one with for example CONTRIBUTING.md) we consider the configuration directory as the current one.

Note

This behavior allows us to not modify the way we develop PyFunceble.

Travis CI¶

Under Travis CI, we search or initiate the configuration at the directory we are currently located.

Warning

We don’t care about the distribution, as long as the TRAVIS_BUILD_DIR environment variable is set, we search or initiate the configuration in the current directory.

Note

If you want to force the directory where we should work, you can initiate the PYFUNCEBLE_OUTPUT_DIR environment variable with the path where we should work.

Linux and MacOS (Darwin Kernel)¶

Under Linux and MacOS, we look for the following directories in their order. If any configuration directory is found, the system proposes you to install them automatically on the first configuration file.

~/.config/PyFunceble
~/.PyFunceble
${PWD}

Note

If the parent directory does not exist, we move to the next possible location in the given order.

This means that under most Linux distributions and MacOS versions, we consider ~/.config/PyFunceble as the configuration location. But if the ~/.config directory does not exist, we fallback to ~/.PyFunceble as the configuration location.

Windows¶

As mentioned by Pat Altimore’s Blog, we used the Per user configuration files synchronized across domain joined machines via Active Directory Roaming section in order to understand what we should do to find our configuration directory.

Under Windows, we look for the following directories in their order. If any configuration directory is found, the system proposes you to install them automatically on the first configuration file.

%APPDATA%\PyFunceble (environnement variable)
%CD%

Note

%CD% is explained by the set command (set /?):

%CD% - expands to the current directory string.

Note

If the parent directory does not exist, we move to the next possible location in the given order.

This means that under most Windows versions, we consider %APPDATA%\PyFunceble - also know as C:\Users\userName\AppData\Roaming\PyFunceble- as the configuration location. But if the %APPDATA% directory does not exist, we fall back to the current directory as the configuration location.

Custom location¶

Sometimes, you may find yourself in a position where you absolutely do not want PyFunceble to use its default configuration location.

For that reason, if you set your desired configuration location along with the PYFUNCEBLE_OUTPUT_DIR environment variable, we take that location as the (default) configuration location.

Autoconfiguration¶

Sometimes, you may find yourself in a position that you do not or you can’t answer the question which asks you if you would like to install the default configuration file.

For that reason, if you set PYFUNCEBLE_AUTO_CONFIGURATION as an environment variable with what you want an assignment, we do not ask that question. We simply do what we have to do without asking anything.

`adblock`¶

Type: boolean

Default value: False

Description: Enable / disable the adblock format decoding.

Note

If this index is set to True, every time we read a given file, we try to extract the elements that are present.

We basically only decode the adblock format.

Note

If this index is set to False, every time we read a given file, we will consider one line as an element to test.

`auto_continue`¶

Type: boolean

Default value: True

Description: Enable / disable the auto continue system.

`command`¶

Type: string

Default value: ""

Description: Set the command to run before each commit (except the final one).

Note

The parsed command is called only if auto_continue and travis are set to True.

`command_before_end`¶

Type: string

Default value: ""

Description: Set the command to run before the final commit.

Note

The parsed command is called only if auto_continue and travis are set to True.

Note

Understand by final commit the commit which will deliver the last element we have to test.

`custom_ip`¶

Type: string

Default value: "0.0.0.0"

Description: Set the custom IP to use when we generate a line in the hosts file format.

Note

This index has no effect if generate_hosts is set to False.

`days_between_db_retest`¶

Type: integer

Default value: 1

Description: Set the number of day(s) between each retest of the INACTIVE and INVALID elements which are present into inactive_db.json.

Note

This index has no effect if inactive_database is set to False.

`debug`¶

Type: boolean

Default value: False

Description: Enable / disable the generation of debug file(s).

Note

This index has no effect if logs is set to False

Warning

Do not touch this index unless you a have good reason to.

Warning

Do not touch this index unless you have been invited to.

`filter`¶

Type: string

Default value: ""

Description: Set the element to filter.

Note

This index should be initiated with a regular expression.

`generate_hosts`¶

Type: boolean

Default value: True

Description: Enable / disable the generation of the hosts file(s).

`generate_json`¶

Type: boolean

Default value: False

Description: Enable / disable the generation of the JSON file(s).

`header_printed`¶

Type: boolean

Default value: False

Description: Say to the system if the header has been already printed or not.

Warning

Do not touch this index unless you have a good reason to.

`hierarchical_sorting`¶

Type: boolean

Default value: False

Description: Say to the system if we have to sort the list and the outputs in a hierarchical order.

`iana_whois_server`¶

Type: string

Default value: whois.iana.org

Description: Set the server to call to get the whois referer of a given element.

Note

This index is only used when generating the iana-domains-db.json file.

Warning

Do not touch this index unless you a have good reason to.

`idna_conversion`¶

Type: boolean

Default value: False

Description: Tell the system to convert all domains to IDNA before testing.

Note

We use domain2idna for the conversion.

`inactive_database`¶

Type: boolean

Default value: True

Description: Enable / Disable the usage of a database to store the INACTIVE and INVALID element to retest overtime.

`less`¶

Type: boolean

Default value: True

Description: Enable / Disable the output of every information of screen.

`local`¶

Type: boolean

Default value: False

Description: Enable / Disable the execution of the test(s) in a local or private network.

`logs`¶

Type: boolean

Default value: True

Description: Enable / Disable the output of all logs.

`mining`¶

Type: boolean

Default value: True

Description: Enable / Disable the mining subsystem.

`no_files`¶

Type: boolean

Default value: False

Description: Enable / Disable the generation of any file(s).

`no_whois`¶

Type: boolean

Default value: False

Description: Enable / Disable the usage of whois in the tests.

`plain_list_domain`¶

Type: boolean

Default value: False

Description: Enable / Disable the generation of the plain list of elements sorted by statuses.

Warning

Do not touch this index unless you a have good reason to.

`quiet`¶

Type: boolean

Default value: False

Description: Enable / Disable the generation of output on the screen.

`referer`¶

Type: string

Default value: ""

Description: Set the referer of the element that is currently under test.

Warning

Do not touch this index unless you a have good reason to.

`seconds_before_http_timeout`¶

Type: integer

Default value: 3

Description: Set the timeout to apply to every HTTP status code request.

Note

This index must be a multiple of 3.

`show_execution_time`¶

Type: boolean

Default value: False

Description: Enable / disable the output of the execution time.

`show_percentage`¶

Type: boolean

Default value: True

Description: Enable / disable the output of the percentage of each status.

`simple`¶

Type: boolean

Default value: False

Description: Enable / disable the simple output mode.

Note

If this index is set to True, the system will only return the result inf format: tested.element STATUS.

`split`¶

Type: boolean

Default value: True

Description: Enable / disable the split of the results files.

Note

Understand with “results files” the mirror of what is shown on screen.

`syntax`¶

Type: boolean

Default value: False

Description: Enable / disable the syntax (only) testing.

Warning

If this index is set to True, we ONLY check for syntax, not availability.

`travis`¶

Type: boolean

Default value: False

Description: Enable / disable the Travis CI autosaving system.

Warning

Do not activate this index unless you are using PyFunceble under Travis CI.

`travis_autosave_commit`¶

Type: string

Default value: "PyFunceble - AutoSave"

Description: Set the default commit message we want to use when have to commit (save) but our tests are not yet completed.

`travis_autosave_final_commit`¶

Type: string

Default value: "PyFunceble - Results"

Description: Set the default final commit message we want to use when we all tests are finished.

`travis_autosave_minutes`¶

Type: integer

Default value: 15

Description: Set the minimum of minutes we have to run before to automatically save our test results.

Note

As many services are setting a rate limit per IP, it’s a good idea to set this value between 1 and 15 minutes.

`travis_branch`¶

Type: string

Default value: master

Description: Set the git branch where we are going to push our results.

`unified`¶

Type: boolean

Default value: False

Description: Enable / Disable the generation of the unified results.

Note

This index has no effect if split is set to True.

`user_agent`¶

Type: string

Default value: "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"

Description: Set the User-Agent to use every time we are requesting something from a web server other than our API.

`verify_ssl_certificate`¶

Type: boolean

Default value: False

Description: Enable / Disable the verification of the SSL/TLS certificate when testing for URL.

Warning

If you set this index to True, you may get false positive result.

Indeed if the certificate is not registered to the CA or is simply invalid and the domain is still alive, you will always get INACTIVE as output.

`whois_database`¶

Type: boolean

Default value: True

Description: Enable / Disable the usage of the whois database to avoid/bypass whois server requests rate limit.

`outputs`¶

Type: dict

Description: Set the needed output tree/names.

Warning

If you choose to change anything please consider deleting our output/ directory and the dir_structure*.json files.

`outputs[default_files]`¶

Type: dict

Description: Set the default name of some important files.

`outputs[default_files][dir_structure]`¶

Type: string

Default value: dir_structure.json

Description: Set the default filename of the file which has the structure to re-construct.

Note

This index has no influence with dir_structure_production.json

`outputs[default_files][iana]`¶

Type: string

Default value: iana-domains-db.json

Description: Set the default filename of the file which has the formatted copy of the IANA root zone database.

`outputs[default_files][inactive_db]`¶

Type: string

Default value: inactive_db.json

Description: Set the default filename of the file which will save the list of elements to retest overtime.

`outputs[default_files][results]`¶

Type: string

Default value: results.txt

Description: Set the default filename of the file which will save the formatted copy of the public suffix database.

`outputs[default_files][public_suffix]`¶

Type: string

Default value: public-suffix.json

Description: Set the default filename of the file which will save the mirror of what is shown on screen.

`outputs[domains]`¶

Type: dict

Description: Set the default name of some important files related to the plain_list_domain index.

`outputs[domains][directory]`¶

Type: string

Default value: domains/

Description: Set the default directory where we have to save the plain list of elements for each status.

`outputs[domains][filename]`¶

Type: string

Default value: list

Description: Set the default filename of the file which will save the plain list of elements.

`outputs[hosts]`¶

Type: dict

Description: Set the default name of some important files related to the generate_hosts index.

`outputs[hosts][directory]`¶

Type: string

Default value: hosts/

Description: Set the default directory where we have to save the hosts files of the elements for each status.

`outputs[hosts][filename]`¶

Type: string

Default value: hosts

Description: Set the default filename of the file which will save the hosts files of the elements.

`outputs[json]`¶

Type: dict

Description: Set the default name of some important files related to the generate_json index.

`outputs[json][directory]`¶

Type: string

Default value: json/

Description: Set the default directory where we have to save the JSON files of the elements for each status.

`outputs[json][filename]`¶

Type: string

Default value: dump.json

Description: Set the default filename of the file which will save the JSON files of the elements.

`outputs[analytic]`¶

Type: dict

Description: Set the default name of some important files and directories related to the generate_hosts index.

`outputs[analytic][directories]`¶

Type: dict

Description: Set the default name of some important directories related to the http_codes[active] index.

`outputs[analytic][directories][parent]`¶

Type: string

Default value: Analytic/

Description: Set the default directory where we are going to put everything related to the HTTP analytic.

`outputs[analytic][directories][potentially_down]`¶

Type: string

Default value: POTENTIALLY_INACTIVE/

Description: Set the default directory where we are going to put all potentially inactive data.

`outputs[analytic][directories][potentially_up]`¶

Type: string

Default value: POTENTIALLY_INACTIVE/

Description: Set the default directory where we are going to put all potentially active data.

`outputs[analytic][directories][up]`¶

Type: string

Default value: POTENTIALLY_INACTIVE/

Description: Set the default directory where we are going to put all active data.

`outputs[analytic][directories][suspicious]`¶

Type: string

Default value: SUSPICIOUS/

Description: Set the default directory where we are going to put all suspicious data.

`outputs[analytic][filenames]`¶

Type: dict

Description: Set the default name of some important files related to the http_codes[active] index and the HTTP analytic subsystem.

`outputs[analytic][filenames][potentially_down]`¶

Type: string

Default value: down_or_potentially_down

Description: Set the default filename where we are going to put all potentially inactive data.

`outputs[analytic][filenames][potentially_up]`¶

Type: string

Default value: potentially_up

Description: Set the default filename where we are going to put all potentially active data.

`outputs[analytic][filenames][up]`¶

Type: string

Default value: active_and_merged_in_results

Description: Set the default filename where we are going to put all active data.

`outputs[analytic][filenames][suspicious]`¶

Type: string

Default value: suspicious_and_merged_in_results

Description: Set the default filename where we are going to put all suspicious data.

`outputs[logs]`¶

Type: dict

Description: Set the default name of some important files and directories related to the logs index.

`outputs[logs][directories]`¶

Type: dict

Description: Set the default name of some important directories related to the logs index.

`outputs[logs][directories][date_format]`¶

Type: string

Default value: date_format/

Description: Set the default directory where we are going to put everything related to the data when the dates are in the wrong format.

`outputs[logs][directories][no_referer]`¶

Type: string

Default value: no_referer/

Description: Set the default directory where we are going to put everything related to the data when no referer is found.

`outputs[logs][directories][parent]`¶

Type: string

Default value: no_referer/

Description: Set the default directory where we are going to put everything related to the data when no referer is found.

`outputs[logs][directories][percentage]`¶

Type: string

Default value: percentage/

Description: Set the default directory where we are going to put everything related to percentages.

`outputs[logs][directories][whois]`¶

Type: string

Default value: whois/

Description: Set the default directory where we are going to put everything related to whois data.

Note

This is the location of all files when the debug index is set to True.

`outputs[logs][filenames]`¶

Type: dict

Description: Set the default filenames of some important files related to the logs index.

`outputs[logs][filenames][auto_continue]`¶

Type: string

Default value: continue.json

Description: Set the default filename where we are going to put the data related to the auto continue subsystem.

Note

This file is allocated if the auto_continue is set to True.

`outputs[logs][filenames][execution_time]`¶

Type: string

Default value: execution.log

Description: Set the default filename where we are going to put the data related to the execution time.

Note

This file is allocated if the show_execution_time is set to True.

`outputs[logs][filenames][percentage]`¶

Type: string

Default value: percentage.txt

Description: Set the default filename where we are going to put the data related to the percentage.

Note

This file is allocated if the show_percentage is set to True.

`outputs[main]`¶

Type: string

Default value: ""

Description: Set the default location where we have to generate the parent_directory directory and its dependencies.

`outputs[parent_directory]`¶

Type: string

Default value: output/

Description: Set the directory name of the parent directory which will contain all previously nouned directories.

`outputs[splited]`¶

Type: dict

Description: Set the default name of some important files and directory related to the split index.

`outputs[splited][directory]`¶

Type: string

Default value: splited/

Description: Set the default directory name where we are going to put the split data.

`status`¶

Type: dict

Description: Set the needed, accepted and status name.

`status[list]`¶

Type: dict

Description: Set the needed and accepted status name.

Warning

All status should be in lowercase.

`status[list][valid]`¶

Type: list

Default value: ["valid","syntax_valid","valid_syntax"]

Description: Set the accepted VALID status.

Note

This status is only shown if the syntax index is activated.

`status[list][up]`¶

Type: list

Default value: ["up","active"]

Description: Set the accepted ACTIVE status.

`status[list][generic]`¶

Type: list

Default value: ["generic"]

Description: Set the accepted generic status.

Note

This status is the one used to say the system that we have to print the complete information on the screen.

`status[list][http_active]`¶

Type: list

Default value: ["http_active"]

Description: Set the accepted status for the outputs[analytic][filenames][up] index.

`status[list][down]`¶

Type: list

Default value: ["down","inactive", "error"]

Description: Set the accepted status INACTIVE index.

`status[list][invalid]`¶

Type: list

Default value: ["ouch","invalid"]

Description: Set the accepted status INVALID index.

`status[list][potentially_down]`¶

Type: list

Default value: ["potentially_down", "potentially_inactive"]

Description: Set the accepted status for the outputs[analytic][filenames][potentially_down] index.

`status[list][potentially_up]`¶

Type: list

Default value: ["potentially_up", "potentially_active"]

Description: Set the accepted status for the outputs[analytic][filenames][potentially_up] index.

`status[list][suspicious]`¶

Type: list

Default value: ["strange", "hum", "suspicious"]

Description: Set the accepted status for the outputs[analytic][filenames][suspicious] index.

`status[official]`¶

Type: dict

Description: Set the official status name.

Note

Those status are the ones that are printed on the screen.

Warning

After any changes here please delete dir_structure.json and the output/ directory.

`status[official][up]`¶

Type: string

Default value: ACTIVE

Description: Set the returned status for the ACTIVE case.

`status[official][down]`¶

Type: string

Default value: INACTIVE

Description: Set the returned status for the INACTIVE case.

`status[official][invalid]`¶

Type: string

Default value: INVALID

Description: Set the returned status for the INVALID case.

`status[official][valid]`¶

Type: string

Default value: VALID

Description: Set the returned status for the VALID case.

Note

This status is only shown if the syntax index is activated.

`http_codes`¶

Type: dict

Description: Handle the interpretation of each status codes when we do and generate our analytic data.

`http_codes[active]`¶

Type: boolean

Default value: True

Description: Enable / Disable the usage of the HTTP status code extraction.

`http_codes[list]`¶

Type: dict

Description: Categorize the http status code as mentioned in the documentation related to the HTTP Code column.

`http_codes[list][up]`¶

Type: list
Default value:
- 100
- 101
- 200
- 201
- 202
- 203
- 204
- 205
- 206
Description: List the HTTP status codes which are considered as ACTIVE.

`http_codes[list][potentially_down]`¶

Type: list
Default value:
- 400
- 402
- 403
- 404
- 409
- 410
- 412
- 414
- 415
- 416
Description: List the HTTP status code which are considered as INACTIVE or POTENTIALLY_INACTIVE.

`http_codes[list][potentially_up]`¶

Type: list
Default value:
- 000
- 300
- 301
- 302
- 303
- 304
- 305
- 307
- 403
- 405
- 406
- 407
- 408
- 411
- 413
- 417
- 500
- 501
- 502
- 503
- 504
- 505
Description: List the HTTP status code which are considered as ACTIVE or POTENTIALLY_ACTIVE.

`links`¶

Type: dict

Description: Set the list of links which can be used/called by the system when needed.

Note

The objective of this index is to avoid hardcoded links when the configuration file is readable.

`links[api_date_format]`¶

Type: string

Default value: https://pyfunceble.funilrys.com/api/date-format

Description: Set the link to use when we share logs.

`links[api_no_referer]`¶

Type: string

Default value: https://pyfunceble.funilrys.com/api/no-referer

Description: Set the link to use when we share logs.

`links[config]`¶

Type: string

Default value: https://raw.githubusercontent.com/funilrys/PyFunceble/master/.PyFunceble_production.yaml

Description: Set the upstream link to the configuration file.

`links[dir_structure]`¶

Type: string

Default value: https://raw.githubusercontent.com/funilrys/PyFunceble/master/dir_structure_production.json

Description: Set the upstream link to the directory structure dump file.

`links[iana]`¶

Type: string

Default value: https://raw.githubusercontent.com/funilrys/PyFunceble/master/iana-domains-db.json

Description: Set the upstream link to the IANA zone file configuration file.

`links[repo]`¶

Type: string

Default value: https://github.com/funilrys/PyFunceble

Description: Set the upstream link to the repository.

`links[requirements]`¶

Type: string

Default value: https://raw.githubusercontent.com/funilrys/PyFunceble/master/requirements.txt

Description: Set the upstream link to the requirements.txt file.

`links[psl]`¶

Type: string

Default value: https://raw.githubusercontent.com/funilrys/PyFunceble/master/public-suffix.json

Description: Set the upstream link to the public suffix database file.

`counter`¶

Type: dict

Description: Setup the internal counter.

Warning

The following is not intended for modification. Exception for debugging or special cases which requires an initiated counter.

`counter[number]`¶

Type: dict

Description: Setup the internal counter.

Warning

The following is not intended for modification. Exception for debugging or special cases which requires an initiated counter.

`counter[number][down]`¶

Type: integer

Default value: 0

Description: Setup the internal down counter.

Warning

The following is not intended for modification. Exception for debugging or special cases which requires an initiated counter.

`counter[number][invalid]`¶

Type: integer

Default value: 0

Description: Setup the internal invalid counter.

Warning

The following is not intended for modification. Exception for debugging or special cases which requires an initiated counter.

`counter[number][tested]`¶

Type: integer

Default value: 0

Description: Setup the internal tested counter.

Warning

The following is not intended for modification. Exception for debugging or special cases which requires an initiated counter.

`counter[number][up]`¶

Type: integer

Default value: 0

Description: Setup the internal up counter.

Warning

The following is not intended for modification. Exception for debugging or special cases which requires an initiated counter.

`counter[percentage]`¶

Type: dict

Description: Setup the internal percentage counter.

Warning

The following is not intended for modification. Exception for debugging or special cases which requires an initiated counter.

`counter[percentage][down]`¶

Type: integer

Default value: 0

Description: Setup the internal down percentage counter.

Warning

The following is not intended for modification. Exception for debugging or special cases which requires an initiated counter.

`counter[percentage][invalid]`¶

Type: integer

Default value: 0

Description: Setup the internal invalid percentage counter.

Warning

The following is not intended for modification. Exception for debugging or special cases which requires an initiated counter.

`counter[percentage][up]`¶

Type: integer

Default value: 0

Description: Setup the internal up percentage counter.

Warning

The following is not intended for modification. Exception for debugging or special cases which requires an initiated counter.

Usage¶

From a terminal¶

Detailed¶

Note

False stand for deactivated when True stand for activated.

`-ad` | `--adblock`¶

Switch the decoding of the adblock format.

Default value: False

If this argument is activated the system will extract all domains or IP from the given adblock file.

`-a` | `--all`¶

Output all available information on the screen.

Default value: False

When activated:

Domain                                                                                               Status      Expiration Date   Source     HTTP Code
---------------------------------------------------------------------------------------------------- ----------- ----------------- ---------- ----------
pyfunceble.readthedocs.io                                                                            ACTIVE      Unknown           NSLOOKUP   302

When deactivated:

Domain                                                                                               Status      HTTP Code
---------------------------------------------------------------------------------------------------- ----------- ----------
pyfunceble.readthedocs.io                                                                            ACTIVE      302

`-c` | `--auto-continue` | `--continue`¶

Switch the value of the auto continue mode.

Default value: True

This argument activates or deactivates the auto-continue subsystem. Indeed, as we can automatically continue if the script has been stopped, this switch allows us to disable or enable the usage of that specific subsystem.

`--clean`¶

Clean all files under output.

As it is sometimes needed to clean our output/ directory, this argument does the job automatically.

Warning

This argument delete everything which are .keep or .gitignore

`--clean-all`¶

Clean all files under output and all file generated by PyFunceble.

Warning

This really deletes everything we generated without any warning.

`--cmd` “something”¶

Pass a command before each commit (except the final one).

Default value: ''

Note

In this example, something should be a script or a program which have to be executed when we reached the end of the given file.

Note

This argument is only used if --travis or travis : true (under .PyFunceble.yaml) are activated.

`--cmd-before-end "something"`¶

Pass a command before the results (final) commit under the Travis mode.

Default value: ''

Note

In this example, something should be a script or a program which have to be executed when we reached the end of the given file.

Note

This argument is only used if --travis or travis : true (under .PyFunceble.yaml) are activated.

`--commit-autosave-message "something"`¶

Replace the default autosave commit message.

Default value: PyFunceble - AutoSave

This argument allows us to set a custom commit message which is going to be used as commit message when saving.

Note

This argument is only used if --travis or travis : true (under .PyFunceble.yaml) are used.

Note

This argument is only used if we have to split the work into multiple processes because a list is too long or the timeout is reached.

Warning

Please avoid the usage of [ci skip] here.

`--commit-results-message "something"`¶

Replace the default results (final) commit message.

Default value: PyFunceble - Results

Note

This argument is only used if --travis or travis : true (under .PyFunceble.yaml) are used.

Note

This argument is only used if we reached the end of the list we are or have to test.

`-d "something"` | `--domain "something"`¶

Set and test the given domain.

This argument will test and give the results of the tests of the given domain.

Note

For this argument (and only for this argument), we are converting the given string to lowercase.

`-db` | `--database`¶

Switch the value of the usage of a database to store inactive domains of the currently tested list.

Default value: True

This argument will disable or enable the usage of a database which saves all INACTIVE and INVALID domain of the given file over time.

Note

The database is retested every x day(s), where x is the number set in -dbr "something".

`-dbr "something"`¶

Set the numbers of days between each retest of domains present into the database of INACTIVE and INVALID domains.

Default value: 1

Note

This argument is only used if -db or inactive_database : true (under .PyFunceble.yaml) are activated.

`--debug`¶

Switch the value of the debug mode.

Default value: False

This argument activates the debug mode. Under the debug mode, everything caught by the whois subsystem is saved.

Warning

Do not use this argument unless you have been told to.

`--directory-structure`¶

Generate the directory and files that are needed and which does not exist in the current directory.

Want to start without anything? This argument generates the output directory automatically for you!

Note

In case of a file or directory not found issue, it’s recommended to remove the dir_structure.json along with the output/ directory before using this argument.

`-ex` | `--execution`¶

Switch the default value of the execution time showing.

Default value: False

Want to know the execution time of your test? Well, this argument will let you know!

`-f "something"` | `--file "something"`¶

Read the given file and test all domains inside it. If a URL is given we download and test the content of the given URL.

Note

We consider one line as one domain or one commented line. A line can be commented at the end.

Note

You can give a raw link and the system will download and test its content.

`--filter "something"`¶

Domain to filter (regex).

Want to test all blogspot from your list? This argument allows you to do that!

Note

This argument should be a regex expression.

`--help`¶

Show the help message and exit.

`-h` | `--host`¶

Switch the value of the generation of hosts file.

Default value: True

This argument will let the system know if it has to generate the hosts file version of each status.

`--hierarchical`¶

Switch the value of the hierarchical sorting of the tested file.

Default value: True

This argument will let the system know if we have to sort the list and our output in hierarchical order.

`--http`¶

Switch the value of the usage of HTTP code.

Default value: True

You don’t want to take the result of the HTTP code execution in consideration? This argument allows you to disable that!

Note

If activated the subsystem will bypass the HTTP status code extraction logic-representation.rst

`--iana`¶

Update/Generate iana-domains-db.json.

This argument generates or updates iana-domains-db.json.

`--idna`¶

Switch the value of the IDNA conversion.

Default value: False

This argument allows the conversion of the domains using domain2idna

Warning

This feature is not supported yet for the URL testing.

`-ip "something"`¶

Change the IP to print with the hosts files.

Default value: 0.0.0.0

`--json`¶

Switch the value of the generation of the JSON formatted list of domains.

Default value: False

`--less`¶

When activated:

Domain                                                                                               Status      HTTP Code
---------------------------------------------------------------------------------------------------- ----------- ----------
pyfunceble.readthedocs.io                                                                            ACTIVE      302

When deactivated:

Domain                                                                                               Status      Expiration Date   Source     HTTP Code
---------------------------------------------------------------------------------------------------- ----------- ----------------- ---------- ----------
pyfunceble.readthedocs.io                                                                            ACTIVE      Unknown           NSLOOKUP   302

`--local`¶

Switch the value of the local network testing.

Default value: False

Want to run a test over a local or private network? This argument will disable the limitation which does not apply to private networks.

`--link "something"`¶

Download and test the given file.

Want to test a raw link? This argument will download and test the given raw link.

`-m` | `--mining`¶

Switch the value of the mining subsystem usage.

Default value: False

Want to find domain or URL linked to a domain in your list? This argument will exactly do that.

`-n` | `--no-files`¶

Switch the value the production of output files.

Default value: False

Want to disable the production of the outputted files? This argument is for you!

`-nl` | `--no-logs`¶

Switch the value of the production of logs files in the case we encounter some errors.

Default value: False

Don’t want any logs to go out of PyFunceble? This argument disables every logs subsystem.

`-nu` | `--no-unified`¶

Switch the value of the production unified logs under the output directory.

Default value: True

This argument disables the generation of result.txt.

`-nw` | `--no-whois`¶

Switch the value the usage of whois to test domain’s status.

Default value: False

Don’t want to use or take in consideration the results from whois? This argument allows you to disable it!

`-p` | `--percentage`¶

Switch the value of the percentage output mode.

Default value: True

This argument will disable or enable the generation of the percentage of each status.

`--plain`¶

Switch the value of the generation of the plain list of domains.

Default value: False:

Want to get a list with all domain for each status? The activation of this argument does the work while testing!

`--production`¶

Prepare the repository for production.

Warning

Do not use this argument unless you have been told to, you prepare a Pull Request or you want to distribute your modified version of PyFunceble.

`-psl` | `--public-suffix`¶

Update/Generate public-suffix.json.

This argument will generate or update public-suffix.json.

`-q` | `--quiet`¶

Run the script in quiet mode.

Default value: False

You prefer to run a program silently? This argument is for you!

`-s` | `--simple`¶

Switch the value of the simple output mode.

Default value: False

Want as less as possible data on screen? This argument returns as less as possible on screen!

`--split`¶

Switch the value of the split of the generated output

Default value: True

Want to get the logs (copy of what you see on screen) on different files? This argument is suited to you!

`--syntax`¶

Switch the value of the syntax test mode.

Default value: False

`-t "something"` | `--timeout "something"`¶

Switch the value of the timeout.

Default value: 3

This argument will set the default timeout to apply everywhere it is possible to set a timeout.

`--travis`¶

Switch the value of the Travis mode.

Default value: False

Want to use PyFunceble under Travis CI? This argument is suited for your need!

`-url "something"` | `--url "something"`¶

Analyze the given URL.

Want to test the availability or an URL? Enjoy this argument!

Note

When we test the availability of an URL, we check the HTTP status code of the given URL.

`-uf "something"` | `--url-file "something"`¶

Read and test the list of URL of the given file. If a URL is given we download and test the content of the given URL.

Note

We consider one line as one URL to test.

Note

You can give a raw link and the system will download and test its content.

`-ua "something"` | `--user-agent "something"`¶

Set the user-agent to use and set every time we interact with everything which is not our logs sharing system.

`-v` | `--version`¶

Show the version of PyFunceble and exit.

`-vsc` | `--verify-ssl-certificate`¶

Switch the value of the verification of the SSL/TLS certificate when testing for URL.

Default value: False

Warning

If you activate the verification of the SSL/TLS certificate, you may get false positive results.

Indeed if the certificate is not registered to the CA or is simply invalid and the domain is still alive, you will always get INACTIVE as output.

`-wdb` | `--whois-database`¶

Switch the value of the usage of a database to store whois data in order to avoid whois servers rate limit.

Default value: True

Global overview¶

usage: PyFunceble [-ad] [-a] [-c] [--autosave-minutes AUTOSAVE_MINUTES]
                [--clean] [--clean-all] [--cmd CMD]
                [--cmd-before-end CMD_BEFORE_END]
                [--commit-autosave-message COMMIT_AUTOSAVE_MESSAGE]
                [--commit-results-message COMMIT_RESULTS_MESSAGE]
                [-d DOMAIN] [-db] [-dbr DAYS_BETWEEN_DB_RETEST] [--debug]
                [--directory-structure] [-ex] [-f FILE] [--filter FILTER]
                [--help] [--hierarchical] [-h] [--http] [--iana] [--idna]
                [-ip IP] [--json] [--less] [--local] [--link LINK] [-m] [-n]
                [-nl] [-nu] [-nw] [-p] [--plain] [--production] [-psl] [-q]
                [--share-logs] [-s] [--split] [--syntax] [-t TIMEOUT]
                [--travis] [--travis-branch TRAVIS_BRANCH] [-u URL]
                [-uf URL_FILE] [-ua USER_AGENT] [-v] [-vsc] [-wdb]

optional arguments:
    -ad, --adblock        Switch the decoding of the adblock format.
                            Configured value: False
    -a, --all             Output all available informations on screen.
                            Configured value: True
    -c, --auto-continue, --continue
                            Switch the value of the auto continue mode.
                            Configured value: True
    --autosave-minutes AUTOSAVE_MINUTES
                            Update the minimum of minutes before we start
                            committing to upstream under Travis CI.
                            Configured value: 15
    --clean               Clean all files under output.
    --clean-all           Clean all files under output and all file generated by
                            PyFunceble.
    --cmd CMD             Pass a command to run before each commit (except the
                            final one) under the travis mode. Configured
                            value: ''
    --cmd-before-end CMD_BEFORE_END
                            Pass a command to run before the results (final)
                            commit under the Travis mode. Configured
                            value: ''
    --commit-autosave-message COMMIT_AUTOSAVE_MESSAGE
                            Replace the default autosave commit message.
                            Configured value: 'PyFunceble -
                            AutoSave'
    --commit-results-message COMMIT_RESULTS_MESSAGE
                            Replace the default results (final) commit message.
                            Configured value: 'PyFunceble -
                            Results'
    -d DOMAIN, --domain DOMAIN
                            Set and test the given domain.
    -db, --database       Switch the value of the usage of a database to store
                            inactive domains of the currently tested list.
                            Configured value: True
    -dbr DAYS_BETWEEN_DB_RETEST, --days-between-db-retest DAYS_BETWEEN_DB_RETEST
                            Set the numbers of days between each retest of
                            domains present into inactive-db.json.
                            Configured value: 1
    --debug               Switch the value of the debug mode.
                            Configured value: False
    --directory-structure
                            Generate the directory and files that are needed and
                            which does not exist in the current directory.
    -ex, --execution      Switch the default value of the execution time
                            showing. Configured value: False
    -f FILE, --file FILE  Read the given file and test all domains inside it. If
                            a URL is given we download and test the content of the
                            given URL.
    --filter FILTER       Domain to filter (regex).
    --help                Show this help message and exit.
    --hierarchical        Switch the value of the hierarchical sorting of tested
                            file. Configured value: True
    -h, --host            Switch the value of the generation of hosts file.
                            Configured value: True
    --http                Switch the value of the usage of HTTP code.
                            Configured value: True
    --iana                Update/Generate `iana-domains-db.json`.
    --idna                Switch the value of the IDNA conversion.
                            Configured value: False
    -ip IP                Change the IP to print in the hosts files with the given one.
                            Configured value: '0.0.0.0'
    --json                Switch the value of the generation of the json list of
                            domain. Configured value: False
    --less                Output less informations on screen.
                            Configured value: False
    --local               Switch the value of the local network testing.
                            Configured value: True
    --link LINK           Download and test the given file.
    -m, --mining          Switch the value of the mining subsystem usage.
                            Configured value: False
    -n, --no-files        Switch the value of the production of output files.
                            Configured value: False
    -nl, --no-logs        Switch the value of the production of logs files in
                            the case we encounter some errors. Configured
                            value: False
    -nu, --no-unified     Switch the value of the production unified logs under
                            the output directory. Configured value:
                            True
    -nw, --no-whois       Switch the value the usage of whois to test domain's
                            status. Configured value: False
    -p, --percentage      Switch the value of the percentage output mode.
                            Configured value: True
    --plain               Switch the value of the generation of the plain list
                            of domain. Configured value: False
    --production          Prepare the repository for production.
    -psl, --public-suffix
                            Update/Generate `public-suffix.json`.
    -q, --quiet           Run the script in quiet mode. Configured
                            value: False
    --share-logs          Switch the value of the sharing of logs.
                            Configured value: True
    -s, --simple          Switch the value of the simple output mode.
                            Configured value: False
    --split               Switch the value of the split of the generated output
                            files. Configured value: True
    --syntax              Switch the value of the syntax test mode.
                            Configured value: False
    -t TIMEOUT, --timeout TIMEOUT
                            Switch the value of the timeout. Configured
                            value: 3
    --travis              Switch the value of the Travis mode.
                            Configured value: False
    --travis-branch TRAVIS_BRANCH
                            Switch the branch name where we are going to push.
                            Configured value: 'master'
    -u URL, --url URL     Analyze the given URL.
    -uf URL_FILE, --url-file URL_FILE
                            Read and test the list of URL of the given file. If a
                            URL is given we download and test the content of the
                            given URL.
    -ua USER_AGENT, --user-agent USER_AGENT
                            Set the user-agent to use and set every time we
                            interact with everything which is not our logs sharing
                            system.
    -v, --version         Show the version of PyFunceble and exit.
    -vsc, --verify-ssl-certificate
                            Switch the value of the verification of the SSL/TLS
                            certificate when testing for URL. Configured
                            value: False
    -wdb, --whois-database
                            Switch the value of the usage of a database to store
                            whois data in order to avoid whois servers rate limit.
                            Configured value: True

Crafted with ♥ by Nissar Chababy (Funilrys) with the
help of https://pyfunceble.rtfd.io/en/master/contributors.html &&
https://pyfunceble.rtfd.io/en/master/special-thanks.html

From a Python script or module¶

Before continuing reading this part, You should know that I consider that you can speak in Python. If it’s not the case, well, it’s the time to learn Python!

As PyFunceble is written in Python, it can be easily imported and used inside a script. This part will represent a basic example of usage.

Basic example¶

"""
This is a basic example which prints one of the official output of PyFunceble.

Note:
* Official output: ACTIVE, INACTIVE, INVALID
"""

from PyFunceble import test as PyFunceble
from PyFunceble import url_test as PyFuncebleURL

print(PyFunceble(domain='google.com'))
print(PyFuncebleURL(url='https://google.com'))

"""
This is a basic example which checks syntax.
"""

from PyFunceble import syntax_check as PyFuncebleDomainSyntax
from PyFunceble import url_syntax_check as PyFuncebleURLSyntax
from PyFunceble import ipv4_syntax_check as PyFuncebleIPv4Syntax

print("google.com", PyFuncebleDomainSyntax(domain="google.com"))
print("https://google.com", PyFuncebleURLSyntax(url="https://google.com"))
print("216.58.207.46", PyFuncebleIPv4Syntax(ip="216.58.207.46"))

print("forest-jump", PyFuncebleDomainSyntax(domain="forest-jump"))
print("https://forest-jump", PyFuncebleURLSyntax(url="https://forest-jump"))
print("257.58.207.46", PyFuncebleIPv4Syntax(ip="257.58.207.46"))

Loop example¶

This part is unnecessary but we wanted to document it!!

"""
This is a loop example which tests a list of domain and processes some action
    according to one of the official output of PyFunceble.

Note:
* Official output: ACTIVE, INACTIVE, INVALID
* You should always use PyFunceble().test() as it's the method which is especially
    suited for `__name__ != '__main__'` usage.
"""
from PyFunceble import test as PyFunceble
from PyFunceble import url_test as PyFuncebleURL

DOMAINS = ["twitter.com", "google.com", "github.com", "github.comcomcom", "funilrys.co"]


def domain_status(domain_or_ip):
    """
    Check the status of the given domain name or IP.

    Argument:
        - domain_or_ip: str
            The domain or IPv4 to test.

    Returns: str
        The status of the domain.
    """

    return PyFunceble(domain_or_ip)


def url_status(url):
    """
    Check the status of the given url.

    Argument:
        - url: str
            The URL to test.

    Returns: str
        The status of the URL.
    """

    return PyFuncebleURL(url)


for domain in DOMAINS:
    print(
        "%s is %s and %s is %s"
        % (
            domain,
            domain_status(domain),
            "http://" + domain,
            url_status("http://" + domain),
        )
    )

Advanced example¶

PyFunceble now allow you to get the following information as a dictionary. The objective behind this feature is to let you know more about the element you are testing.

{
    "tested": None, # The tested element.
    "expiration_date": None, # The expiration_date of the element if found.
    "domain_syntax_validation": None, # The domain syntax validation status.
    "http_status_code": None, # The status code of the tested element.
    "ip4_syntax_validation": None, # The IPv4 syntax validation status.
    "nslookup": [], # A list of IP of the tested element.
    "status": None, # The status matched by PyFunceble.
    "url_syntax_validation": None, # The url syntax validation status.
    "whois_server": None, # The whois server if found.
    "whois_record": None, # The whois record if whois_server is found.
}

To get that information simply work with our interface like follow :)

"""
This is an advanced example which prints some information about the tested element.

Note:
* Official output: ACTIVE, INACTIVE, INVALID
"""

from PyFunceble import test as PyFunceble
from PyFunceble import url_test as PyFuncebleURL

domain_testing = PyFunceble(domain='google.com', complete=True)
url_testing = PyFuncebleURL(url='https://google.com', complete=True)

print(domain_testing['nslookup'])
print(domain_testing['domain_syntax_validation'])
print(domain_testing['domain'], domain_testing['status'])

print(url_testing['nslookup'])
print(url_testing['domain_syntax_validation'])
print(url_testing['domain'], domain_testing['status'])

From Travis CI¶

As we offer an argument named --travis to activate the usage of PyFunceble in a Travis CI instance, we document here what you need to know!

Configuration¶

Note

This part only present a commented .travis.yml so that you can understand where to start.

If you need more practical examples, feel free to report to one of Dead-Hosts repositories which use PyFunceble with Travis CI.

env:
    global:
        # The following is your encrypted GitHub API key.
        # Indeed as we are going to push to the repository, this is needed.
        - secure: QQdKFquFFojFT9XJ1XZp4EMoDTVoXFgqZq8XU+sCVf+pJQR6d/oKBp8rnSTCnZizWOQXUjGXUUxUpSG/dYGyBLjo3rH3rsn9ciZHVfubxbwK860w4sqibl4DvhCv2rdsFtvzXnhm4P9OL3i+krKdewh9fxpNyUU58qOgfnS7mK9FcFhb8z5ak2sxU2XRZedwm6Ro0oyVKs8kFkL4YaADfNyAHlGTfr9rVmE52WXQXQENktb9gFgR2A8ZnmLy0BCMZGkPDShJnjRDWD4DErtasLmLQvWpzOBwdbVJTY6U9KDRXVNdC9lp5E5Ba/dc0y36q6vjfgJR+QchetOtHgNbKYbLB8c26Di90OZCFJsxMNcl1Wct4qFPXkFGvjXrISW6pbdPL5Plto0Ig3iLiulhYOPVArysMIk9ymtSXP+WE7VWX01LQ1fEkIoSfeVZ2caTnCmTsoHVGRRe978CojKaT7yU45kb15hcyDrzptQ8EP2hfxeh5F7KtueQ6Rsb9LFDZMkMDKflZn6a+bRhESlmWWmYB9stzGzTurQA1E1bcSACJ8A8hG5nHBzZYJ2S+OY0PE7UdyOJ0JK0qe/67d+F9ocQdIoFpDDTdgIjHerQnD2wRg1aKPzLDb4jJTpqgr5ssPrqUAKl3st7gyaAZzCEADPDnIBDjOJS+mFWbx9DKgc=
        # This is the Git name we have to set. (git config user.name)
        - GIT_NAME: Travis CI
        # This is the Git Email we have to set. (git config user.email)
        - GIT_EMAIL: dead-hosts@funilrys.com
        # This is the full slug of the repository we are working with.
        - TRAVIS_REPO_SLUG: dead-hosts/repository-structure
        # This is the branch we have to checkout and push to.
        - GIT_BRANCH: master

# This is the language we use.
language: python

# This is the python version we are going to use for the tests.
# Note: you can add any 3.x version to the list.
python:
- "3.6"

# The following will tell Travis CI to ends as fast as possible.
matrix:
    fast_finish: true

# Here we are setting what Travis CI have to cache.
cache:
    # We are caching pip3 as we use it to install PyFunceble
    - pip3

install:
    # We install the development version of PyFunceble. If you prefer the stable version replace `pyfunceble-dev` with `pyfunceble`.
    - pip3 install pyfunceble-dev

# Our tests start here.
script:
    # Let's say we want our results and our PyFunceble infrastructure to be saved in a directory called `PyFunceble-tests`

    # We move inside it.
    - cd PyFunceble-tests
    # We test the file `my_awesome_list` which is located inside the current directory.
    # Note: we precise the `--travis` argument here,
    #     but you work without it if you set `travis: true` inside your `.PyFunceble.yaml`
    - PyFunceble --travis -f my_awesome_list --plain

# The following initiate email notification logic.
notifications:
    # As we want to get a mail on failure and on status change, we set the following.
    on_success:   change
    on_failure:   always

Getting a GitHub token¶

For the secure index of the .travis.yml file, you have to generate a new GitHub token.

After you got your token, please write it or save it in a safe place as you’re going to need it every time you’re going to interact with Travis CI.

Note

The scope to set is public_repo but you can also set others depending on your needs.

Encrypting the token for Travis CI usage¶

To encrypt the token simply replace and execute the following according to your personal case.

$ travis encrypt 'GH_TOKEN=theGeneratedToken' -r 'The content of TRAVIS_REPO_SLUG' --add

Warning

Please do not execute the following explicitly without replacing theGeneratedToken with your previously generated GitHub token and The content of TRAVIS_REPO_SLUG with your repository slug.

Note

The usage of --add ensure that the travis program automatically add the secure index to the .travis.yml file.

Columns¶

Because PyFunceble provides multiple information in a table, we chose to document them all. The objective of this page is to reflect what the code actually do in a more clear and understandable way.

Domains¶

This column is one of the basic ones as it gives us the name of the last tested domain or IP.

Status¶

There’s 3 possible output for this column.

ACTIVE¶

This status is returned when one of the following cases is met:

We can extract the expiration date from Lookup().whois().
- Please note that we don’t check if the date is in the past.
Lookup().nslookup() don’t return an error.
- Please note that we don’t read the returned value.
HTTPCode().get() return one the following code [100, 101, 200, 201, 202, 203, 204, 205, 206].

INACTIVE¶

This status is returned when all the following cases are met:

We can’t extract the expiration date from Lookup().whois().
Lookup().nslookup() don’t return an error.

INVALID¶

This status is returned when all the following cases are met:

Domain/IP does not match pass our syntax checker.
Domain extension is unregistered in IANA Root Zone Database.

Note

Understand by this that the extension is not present in the iana-domains-db.json file.

VALID¶

This status is returned when we are checking for syntax. It is the equivalent of ACTIVE but for syntax checking.

Expiration Date¶

There’s two possible output for this column.

Unknown¶

Unknown is returned when we could not extract the expiration date from Lookup().whois() outputs.

A date¶

Only if we could extract the expiration date from Lookup().whois(), the date becomes formatted like 02-jan-2017.

Source¶

At this time, there’s 5 possible output for this column.

HTTP Code¶

This source is returned when all the following cases are met:

We can’t extract the expiration date from Lookup().whois().
The INACTIVE status is the one returned by other methods.
HTTPCode().get() outputs is one the following [100, 101, 200, 201, 202, 203, 204, 205, 206].

IANA¶

This source is always returned when the domain has the status INVALID. The usage of this source comes from the comparison of the element extension against the IANA Root Zone Database.

SYNTAX¶

This source is always returned when the domain has the status INVALID or in the case that we are only checking for syntax instead of availability. The usage of this source comes from the comparison of the element against our domain, IP or URL syntax validation system.

NSLOOKUP¶

This source is always returned when the taken decision of the status of the domain/IP comes from Lookup().nslookup() outputs.

SPECIAL¶

As PyFunceble grows, I thought that a bit of filtering for special cases would be great. So I introduced the SPECIAL source.

Note

Please consider all 3 digits number that are listed in this section as the HTTP status code catched by HTTPCode().get().

`.blogspot.`¶

All 404 are returned as INACTIVE
All 301 which are blocked by Google or does not exist are returned as INACTIVE
All 302 which are blocked by Google are returned as INACTIVE

`*.canalblog.com`¶

All 404 are returned as INACTIVE

`*.doubleclick.net`¶

All 404 are returned as INACTIVE

`*.liveadvert.com`¶

All 404 are returned as INACTIVE

`*.skyrock.com`¶

All 404 are returned as INACTIVE

`*.tumblr.com`¶

All 404 are returned as INACTIVE

`*.wordpress.com`¶

All 301 which match doesn’t exist are returned as INACTIVE

IP with range¶

All IPv4 with a range (for example 0.0.0.0/24) are returned as ACTIVE

HTTP Code¶

Note

This section reflects exactly what the code does. So any changes made here should also be reported to the code or at least the configuration file.

Note

A *** in this column means that it was impossible to catch the HTTP status code from the web server.

We have categorized the HTTP Code into 3 parts.

Active
- Consider those ones like the one that influences HTTP source logic.
- Please note that the domain is automatically introduced into the official outputs but we keep a record of it output/Analytic/ACTIVE.
Potentially Active
- If the domain status returned by other methods are different from INACTIVE or INVALID and the HTTP status code is into that list, we save the domain into output/Analytic/POTENTIALLY_ACTIVE
Inactive or potentially inactive
- If the domain status returned by other methods are different from ACTIVE and the HTTP status code is on that list, we save the domain into output/Analytic/POTENTIALLY_INACTIVE

As active¶

100 - Continue
101 - Switching Protocols
200 - OK
201 - Created
202 - Accepted
203 - Non-Authoritative Information
204 - No Content
205 - Reset Content
206 - Partial Content

As potentially active¶

000
300 - Multiple Choices
301 - Moved Permanently
302 - Found
303 - See Other
304 - Not Modified
305 - Use Proxy
307 - Temporary Redirect
403 - Forbidden
405 - Method Not Allowed
406 - Not Acceptable
407 - Proxy Authentication Required
408 - Request Timeout
411 - Length Required
413 - Request Entity Too Large
417 - Expectation Failed
500 - Internal Server Error
501 - Not Implemented
502 - Bad Gateway
503 - Service Unavailable
504 - Gateway Timeout
505 - HTTP Version Not Supported

As inactive or potentially inactive¶

400 - Bad Request
401 - Unauthorized
402 - Payment Required (Not in use but may be seen in the future)
404 - Not Found
409 - Conflict
410 - Gone
412 - Precondition Failed
414 - Request-URI Too Long
415 - Unsupported Media Type
416 - Requested Range Not Satisfiable

Logic representation¶

Don’t have time to review the code and want to know exactly what PyFunceble does in general? This flowchart is there for you!

Note

This is a simplified version of what we actually do.

Note

The diagrams were generated with the help of yEd Graph Editor.

Domain testing¶

IP testing¶

URL testing¶

API documentation¶

Note

This section document what you can call directly when you use PyFunceble as an imported module.

The tool to check the availability or syntax of domains, IPv4 or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

This submodule is the main entry of PyFunceble.

Author:: Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:: https://pyfunceble.readthedocs.io/en/master/special-thanks.html
Contributors:: http://pyfunceble.readthedocs.io/en/master/special-thanks.html
Project link:: https://github.com/funilrys/PyFunceble
Project documentation:: https://pyfunceble.readthedocs.io/en/master/
Project homepage:: https://funilrys.github.io/PyFunceble/

License:

MIT License

Copyright (c) 2017-2018 Nissar Chababy

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

PyFunceble.ipv4_syntax_check(ip)[source]¶

Check the syntax of the given IPv4.

Parameters:	ip (str) – The IPv4 to check the syntax for.
Returns:	The syntax validity.
Return type:	bool

PyFunceble.load_config(under_test=False)[source]¶

Load the configuration.

Parameters:	under_test (bool) – Tell us if we only have to load the configuration file (True) or load the configuration file and initate the output directory if it does not exist (False).

PyFunceble.syntax_check(domain)[source]¶

Check the syntax of the given domain.

Parameters:	domain (str) – The domain to check the syntax for.
Returns:	The syntax validity.
Return type:	bool

PyFunceble.test(domain, complete=False)[source]¶

Test the availability of the given domain or IP.

Parameters:	domain (str) – The domain or IP to test. complete (bool) – Activate the return of a dict with some significant data from the test.
Returns:	The status or the informations of the domain.
Return type:	str\|dict

Note

This function abstract and simplify for the access to the core for the end-user.

PyFunceble.url_syntax_check(url)[source]¶

Check the syntax of the given URL.

Parameters:	url (str) – The URL to check the syntax for.
Returns:	The syntax validity.
Return type:	bool

PyFunceble.url_test(url, complete=False)[source]¶

Test the availability of the given URL.

Parameters:	url (str) – The URL to test. complete (bool) – Activate the return of a dict with some significant data from the test.
Returns:	The status or the informations of the URL.
Return type:	str\|dict

Note

This function abstract and simplify for the access to the core for the end-user.

Code Documentation¶

Note

This section will document every part (except the API section) of our code base.

Helpers¶

Problematic¶

How can we avoid writing the same thing every time?

Documentation¶

The tool to check the availability or syntax of domains, IPv4 or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

This submodule will provide the helpers.

Author:: Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:: https://pyfunceble.readthedocs.io/en/master/special-thanks.html
Contributors:: http://pyfunceble.readthedocs.io/en/master/special-thanks.html
Project link:: https://github.com/funilrys/PyFunceble
Project documentation:: https://pyfunceble.readthedocs.io/en/master/
Project homepage:: https://funilrys.github.io/PyFunceble/

License:

MIT License

Copyright (c) 2017-2018 Nissar Chababy

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

class PyFunceble.helpers.Command(command)[source]¶

Shell command execution.

Parameters:	command (str) – The command to execute

_decode_output(to_decode)[source]¶

Decode the output of a shell command in order to be readable.

Parameters:	to_decode – Output of a command to decode.
Type:	bytes
Returns:	The decoded output.
Return type:	str

execute()[source]¶

Execute the given command.

Returns:	The output of the command.
Return type:	str

class PyFunceble.helpers.Dict(main_dictionnary=None)[source]¶

Dictionary manipulations.

Parameters:	main_dictionnary (dict) – The dict we are working with.

classmethod from_json(data)[source]¶

Convert a JSON formatted string into a dictionary.

Parameters:	data (str) – A JSON formatted string to convert to dict format.
Returns:	The dict representation of the JSON formatted string.
Return type:	dict

classmethod from_yaml(data)[source]¶

Convert a YAML formatted string into a dictionary.

Parameters:	data (str) – A YAML formatted string to convert to dict format.
Returns:	The dict representation of the YAML formatted string.
Return type:	dict

merge(to_merge, strict=True)[source]¶

Merge the content of to_merge into the given main dictionnary.

Parameters:	to_merge (dict) – The dictionnary to merge. strict (bool) – Tell us if we have to strictly merge lists. `True`: We follow index :code`False`: We follow element (content)
Returns:	The merged dict.
Return type:	dict

remove_key(key_to_remove)[source]¶

Remove a given key from a given dictionary.

Parameters:	key_to_remove (list\|str) – The key(s) to delete.
Returns:	The dict without the given key(s).
Return type:	dict\|None

rename_key(key_to_rename, strict=True)[source]¶

Rename the given keys from the given dictionary.

Parameters:	key_to_rename (dict) – The key(s) to rename. Expected format: `{old:new}` strict – Tell us if we have to rename the exact index or the index which looks like the given key(s)
Returns:	The well formatted dict.
Return type:	dict\|None

to_json(destination)[source]¶

Save a dictionnary into a JSON file.

Parameters:	destination (str) – A path to a file where we’re going to write the converted dict into a JSON format.

to_yaml(destination, flow_style=False)[source]¶

Save a dictionnary into a YAML file.

Parameters:	destination (str) – A path to a file where we’re going to write the converted dict into a JSON format.

class PyFunceble.helpers.Directory(directory)[source]¶

Directory manipulation.

Parameters:	directory (str) – A path to the directory to manipulate.

fix_path(splited_path=None)[source]¶

Fix the path of the given path.

Parameters:	splited_path (list) – A list to convert to the right path.
Returns:	The fixed path.
Return type:	str

class PyFunceble.helpers.Download(link, destination=None, return_data=False, verify_certificate=True)[source]¶

Download or return the content of the given link.

Parameters:	link (str) – The link to download. destination (str) – The location where we should save the downloaded content. return_data (bool) – Tell us if we need to return the page content or write its content into the given destination. verify_certificate (bool) – Tell us if we need to verify the SSL/TLS certificate.

text()[source]¶

Download the given link and return or save its requests.text at the given destination.

Return type:	mixed
Raises:	`Exception` If the status code is not `200`.

class PyFunceble.helpers.File(file)[source]¶

File treatment/manipulations.

Parameters:	file (str) – A path to the file to manipulate.

copy(destination)[source]¶

Copy the given file to the destination.

Parameters:	destination (str) – The destination of the copy.

delete()[source]¶: Delete a given file path.

read()[source]¶

Read a given file path and return its content.

Returns:	The content of the given file path.
Return type:	str

write(data_to_write, overwrite=False)[source]¶

Write or append data into the given file path.

Parameters:	data_to_write (str) – The data to write. overwrite (bool) – Tell us if we have to overwrite the content of the file we are working with.

class PyFunceble.helpers.Hash(file_path=None, algorithm='sha512', only_hash=False, data=None)[source]¶

Get and return the hash a file with the given algorithm.

Parameters:	file_path (str) – The path to the file we have to hash. algorithm (str) – The algorithm to use. only_hash – Tell us if we only have to return the desired algorithm instead of the dummy dict format.

Note

The original version can be found at https://git.io/vFQrK.

_hash_data(algo)[source]¶

Get hash of the given data.

Parameters:	algo (str) – The algorithm to use.

_hash_file(algo)[source]¶

Get the hash of the given file

Parameters:	algo (str) – The algorithm to use.
Returns:	The hexdigest of the data.
Return type:	str

get()[source]¶: Return the hash of the given file

class PyFunceble.helpers.List(main_list=None)[source]¶

List manipulation.

Parameters:	main_list (list) – The list to manipulate.

custom_format(key_method, reverse=False)[source]¶

Return a well formatted list. With the key_method as a function/method to format the elements before sorting.

Parameters:	key_method (function\|method) – A function or method to use to format the readed element before sorting. reverse (bool) – Tell us if we have to reverse the list.
Returns:	A sorted list.
Return type:	list

format()[source]¶

Return a well formatted list. Basicaly, it’s sort a list and remove duplicate.

Returns:	A sorted, without duplicate, list.
Return type:	list

merge(to_merge, strict=True)[source]¶

Merge to_merge into the given main list.

Parameters:	to_merge (list) – The list to merge. strict (bool) – Tell us if we have to respect index (True) or not (False).
Returns:	The merged list.
Return type:	list

class PyFunceble.helpers.Regex(data, regex, **args)[source]¶

A simple implementation ot the python.re package

Parameters:

data (str) – The data to check.
regex (str) – The regex to match.
group (int) – The group to return.
rematch (bool) –
Allow to return the matched groups into a formatted list.

Note

This is an implementation of Bash ${BASH_REMATCH}
replace_with (str) – The value to replace the matched regex with.
occurences (int) – The number of occurence(s) to replace.
return_type – Tell us if we have to return the matched data or simply check if we matched (True) or not (False)

match()[source]¶

Used to get exploitable result of re.search

Returns:	The data of the match status.
Return type:	mixed

matching_list()[source]¶: Return a list of the string which match the given regex.

not_matching_list()[source]¶: Return a list of string which don’t match the given regex.

replace()[source]¶

Used to replace a matched string with another.

Returns:	The data after replacement.
Return type:	str

`Download()`¶

class PyFunceble.helpers.Download(link, destination=None, return_data=False, verify_certificate=True)[source]

Download or return the content of the given link.

Parameters:	link (str) – The link to download. destination (str) – The location where we should save the downloaded content. return_data (bool) – Tell us if we need to return the page content or write its content into the given destination. verify_certificate (bool) – Tell us if we need to verify the SSL/TLS certificate.

text()[source]

Download the given link and return or save its requests.text at the given destination.

Return type:	mixed
Raises:	`Exception` If the status code is not `200`.

`Command()`¶

class PyFunceble.helpers.Command(command)[source]

Shell command execution.

Parameters:	command (str) – The command to execute

_decode_output(to_decode)[source]

Decode the output of a shell command in order to be readable.

Parameters:	to_decode – Output of a command to decode.
Type:	bytes
Returns:	The decoded output.
Return type:	str

execute()[source]

Execute the given command.

Returns:	The output of the command.
Return type:	str

`Regex()`¶

class PyFunceble.helpers.Regex(data, regex, **args)[source]

A simple implementation ot the python.re package

Parameters:

data (str) – The data to check.
regex (str) – The regex to match.
group (int) – The group to return.
rematch (bool) –
Allow to return the matched groups into a formatted list.

Note

This is an implementation of Bash ${BASH_REMATCH}
replace_with (str) – The value to replace the matched regex with.
occurences (int) – The number of occurence(s) to replace.
return_type – Tell us if we have to return the matched data or simply check if we matched (True) or not (False)

match()[source]

Used to get exploitable result of re.search

Returns:	The data of the match status.
Return type:	mixed

matching_list()[source]: Return a list of the string which match the given regex.

not_matching_list()[source]: Return a list of string which don’t match the given regex.

replace()[source]

Used to replace a matched string with another.

Returns:	The data after replacement.
Return type:	str

`Dict()`¶

class PyFunceble.helpers.Dict(main_dictionnary=None)[source]

Dictionary manipulations.

Parameters:	main_dictionnary (dict) – The dict we are working with.

classmethod from_json(data)[source]

Convert a JSON formatted string into a dictionary.

Parameters:	data (str) – A JSON formatted string to convert to dict format.
Returns:	The dict representation of the JSON formatted string.
Return type:	dict

classmethod from_yaml(data)[source]

Convert a YAML formatted string into a dictionary.

Parameters:	data (str) – A YAML formatted string to convert to dict format.
Returns:	The dict representation of the YAML formatted string.
Return type:	dict

merge(to_merge, strict=True)[source]

Merge the content of to_merge into the given main dictionnary.

Parameters:	to_merge (dict) – The dictionnary to merge. strict (bool) – Tell us if we have to strictly merge lists. `True`: We follow index :code`False`: We follow element (content)
Returns:	The merged dict.
Return type:	dict

remove_key(key_to_remove)[source]

Remove a given key from a given dictionary.

Parameters:	key_to_remove (list\|str) – The key(s) to delete.
Returns:	The dict without the given key(s).
Return type:	dict\|None

rename_key(key_to_rename, strict=True)[source]

Rename the given keys from the given dictionary.

Parameters:	key_to_rename (dict) – The key(s) to rename. Expected format: `{old:new}` strict – Tell us if we have to rename the exact index or the index which looks like the given key(s)
Returns:	The well formatted dict.
Return type:	dict\|None

to_json(destination)[source]

Save a dictionnary into a JSON file.

Parameters:	destination (str) – A path to a file where we’re going to write the converted dict into a JSON format.

to_yaml(destination, flow_style=False)[source]

Save a dictionnary into a YAML file.

Parameters:	destination (str) – A path to a file where we’re going to write the converted dict into a JSON format.

`List()`¶

class PyFunceble.helpers.List(main_list=None)[source]

List manipulation.

Parameters:	main_list (list) – The list to manipulate.

custom_format(key_method, reverse=False)[source]

Return a well formatted list. With the key_method as a function/method to format the elements before sorting.

Parameters:	key_method (function\|method) – A function or method to use to format the readed element before sorting. reverse (bool) – Tell us if we have to reverse the list.
Returns:	A sorted list.
Return type:	list

format()[source]

Return a well formatted list. Basicaly, it’s sort a list and remove duplicate.

Returns:	A sorted, without duplicate, list.
Return type:	list

merge(to_merge, strict=True)[source]

Merge to_merge into the given main list.

Parameters:	to_merge (list) – The list to merge. strict (bool) – Tell us if we have to respect index (True) or not (False).
Returns:	The merged list.
Return type:	list

`Directory()`¶

class PyFunceble.helpers.Directory(directory)[source]

Directory manipulation.

Parameters:	directory (str) – A path to the directory to manipulate.

fix_path(splited_path=None)[source]

Fix the path of the given path.

Parameters:	splited_path (list) – A list to convert to the right path.
Returns:	The fixed path.
Return type:	str

`File()`¶

class PyFunceble.helpers.File(file)[source]

File treatment/manipulations.

Parameters:	file (str) – A path to the file to manipulate.

copy(destination)[source]

Copy the given file to the destination.

Parameters:	destination (str) – The destination of the copy.

delete()[source]: Delete a given file path.

read()[source]

Read a given file path and return its content.

Returns:	The content of the given file path.
Return type:	str

write(data_to_write, overwrite=False)[source]

Write or append data into the given file path.

Parameters:	data_to_write (str) – The data to write. overwrite (bool) – Tell us if we have to overwrite the content of the file we are working with.

`Hash()`¶

class PyFunceble.helpers.Hash(file_path=None, algorithm='sha512', only_hash=False, data=None)[source]

Get and return the hash a file with the given algorithm.

Parameters:	file_path (str) – The path to the file we have to hash. algorithm (str) – The algorithm to use. only_hash – Tell us if we only have to return the desired algorithm instead of the dummy dict format.

Note

The original version can be found at https://git.io/vFQrK.

_hash_data(algo)[source]

Get hash of the given data.

Parameters:	algo (str) – The algorithm to use.

_hash_file(algo)[source]

Get the hash of the given file

Parameters:	algo (str) – The algorithm to use.
Returns:	The hexdigest of the data.
Return type:	str

get()[source]: Return the hash of the given file

AdBlock¶

Problematic¶

How can we efficiently decode AdBlock filter list?

Documentation¶

class PyFunceble.adblock.AdBlock(list_from_file, aggressive=False)[source]¶

Provide the adblock decoding logic.

Parameters:	list_from_file (list) – The file in list format.

classmethod _extract_base(element)[source]¶

Extract the base of the given element.

Parameters:	element (str\|list) – The element we are working with.

classmethod _format_decoded(to_format, result=None)[source]¶

Format the exctracted adblock line before passing it to the system.

Parameters:	to_format (str) – The extracted line from the file. result (list) – A list of the result of this method.
Returns:	The list of domains or IP to test.
Return type:	list

_handle_options(options)[source]¶

Handle the data from the options.

Parameters:	options (list) – The list of options from the rule.
Returns:	The list of domains to return globally.
Return type:	list

classmethod _is_to_ignore(line)[source]¶

Check if we have to ignore the given line.

Parameters:	line (str) – The line from the file.

_remove_ignored()[source]¶: Removed the ignored element from the given list.

decode()[source]¶

Decode/extract the domains to test from the adblock formated file.

Returns:	The list of domains to test.
Return type:	list

Auto-continue¶

Problematic¶

How can we continue the test after executable stop?

Documentation¶

class PyFunceble.auto_continue.AutoContinue[source]¶

Provide the auto-continue subsystem.

backup()[source]¶: Backup the current execution state.

restore()[source]¶: Restore data from the given path.

Auto-save¶

Note

Only Travis CI is actually supported.

Travis CI problematic¶

How can we bypass the default Travis CI timeout of 45 minutes?

Documentation¶

class PyFunceble.auto_save.AutoSave(is_last_domain=False, is_bypass=False)[source]¶

Provide the autosave logic.

Parameters:	is_last_domain (bool) – Tell this subsystem if we are at the very end of the file testing. is_bypass (bool) – Tell this subsystem if we are in bypassing mode.

_travis()[source]¶: Logic behind autosave under Travis CI.

classmethod travis_permissions()[source]¶: Set permissions in order to avoid issues before commiting.

Check¶

Problematic¶

How can we efficiently check the format of IP, domains, and URL?

Documentation¶

class PyFunceble.check.Check(element=None)[source]¶

Provide a place to check several things around URL, IP or domain.

Parameters:	element (str) – The element (URL, IP or domain) to check.

is_domain_valid(domain=None, subdomain_check=False)[source]¶

Check if the given domain is a valid.

Parameters:	domain (str) – The domain to validate. subdomain_check (bool) – Activate the subdomain checking.
Returns:	The validity of the sub-domain.
Return type:	bool

is_ip_range(ip_to_check=None)[source]¶

Check if the given IP is a valid IPv4.

Parameters:	ip_to_check (str) – The IP to test.
Returns:	The validity of the IP.
Return type:	bool

Note

We only test IPv4 because for now we only them for now.

is_ip_valid(ip_to_check=None)[source]¶

Check if the given IP is a valid IPv4.

Parameters:	ip_to_check (str) – The IP to test.
Returns:	The validity of the IP.
Return type:	bool

Note

We only test IPv4 because for now we only them for now.

is_subdomain(domain=None)[source]¶

Check if the given subdomain is a subdomain.

Parameters:	domain (str) – The domain to validate.
Returns:	The validity of the subdomain.
Return type:	bool

is_url_valid(url=None, return_base=False, return_formatted=False)[source]¶

Check if the given URL is valid.

Parameters:	url (str) – The url to validate. return_base – Allow us the return of the url base (if URL formatted correctly). return_formatted (bool) – Allow us to get the URL converted to IDNA if the conversion is activated.
Returns:	The validity of the URL or its base.
Return type:	bool\|str

Cleaning¶

Problematic¶

How can we clean the output/ directory so we do not have a collision between old and new files?

Documentation¶

class PyFunceble.clean.Clean(list_to_test, clean_all=False)[source]¶

Provide the cleaning logic(s).

Parameters:	list_to_test (list\|None) – The list of domains we are testing. clean_all (bool) – Tell the subsystem if we need to clean all. Which include, of course, the output directory but also all other file(s) generated by our system.

Configuration¶

Problematics¶

How can we avoid the usage of tool.py?
How can we make personalization more simple?

Documentation¶

`Load()`¶

class PyFunceble.config.Load(path_to_config)[source]¶

Help us load the configuration(s) file(s).

Parameters:	path_to_config (str) – The possible path to the configuration to load.

classmethod _install_directory_structure_file()[source]¶: Download the latest version of dir_structure_production.json.

classmethod _install_iana_config()[source]¶: Download iana-domains-db.json if not present.

_install_production_config()[source]¶: Download the production configuration and install it in the current directory.

classmethod _install_psl_config()[source]¶: Download public-suffix.json if not present.

_load_config_file()[source]¶: Load .PyFunceble.yaml into the system.

classmethod _set_path_to_configs(path_to_config)[source]¶

Set the paths to the configuration files.

Parameters:	path_to_config (str) – The possible path to the config to load.
Returns:	The path to the config to read (0), the path to the default configuration to read as fallback.(1)
Return type:	tuple

`Version()`¶

class PyFunceble.config.Version(used=False)[source]¶

Compare the local with the upstream version.

Parameters:	used (bool) – True: Version is configured for simple usage. False: Version compare local with upstream.

classmethod check_versions(local, upstream)[source]¶

Compare the given versions.

Parameters:

local (list) – The local version converted by split_versions().
upstream (list) – The upstream version converted by split_versions().

Returns:

True: local < upstream
None: local == upstream
False: local > upstream

Return type:

bool|None

classmethod check_versions_literally(local, upstream)[source]¶

Compare the given versions literally.

Parameters:

local (str) – The local version converted by split_versions().
upstream (str) – The upstream version converted by split_versions().

Returns:

True: local == upstream
False: local != upstream

Return type:

bool

compare()[source]¶: Compare the current version with the upstream saved version.

classmethod is_cloned()[source]¶: Let us know if we are currently in the cloned version of PyFunceble which implicitly mean that we are in developement mode.

classmethod right_url_from_version(url)[source]¶

Convert the GitHub URL to the right one depending of the branch or version we are working with.

Parameters:	url (str) – The URL to convert.
Returns:	The converted URL.
Return type:	str

classmethod split_versions(version, return_non_digits=False)[source]¶

Convert the versions to a shorter one.

Parameters:	version (str) – The version to split. return_non_digits (bool) – Activate the return of the non-digits parts of the splitted version.
Returns:	The splitted version name/numbers.
Return type:	list

Core¶

Documentation¶

class PyFunceble.core.Core(**args)[source]¶

Main entry to PyFunceble. Brain of the program. Also known as “put everything together to make the system works”.

Parameters:

domain_or_ip_to_test (str) – A domain or IP to test.
file_path (str) – A path to a file to read and test.
url_to_test (str) – A URL to test.
url_file (str) – A path to a file which contains URL to test.
link_to_test (str) – A link to a file to download and test.
modulo_test – If set to True, it will tell the system that we are working as an exported module.
modulo_test – bool

_entry_management()[source]¶: Avoid to have 1 millions line into self.__init__()

_entry_management_url()[source]¶: Manage the loading of the url system.

classmethod _entry_management_url_download(passed)[source]¶

Check if the given information is a URL. If it is the case, it download and update the location of file to test.

Parameters:	passed (str) – The url passed to the system.
Returns:	The state of the check.
Return type:	bool

classmethod _extract_domain_from_file()[source]¶

Extract all non commented lines from the file we are testing.

Returns:	The elements to test.
Return type:	list

_file_decision(current, last, status=None)[source]¶

Manage the database, autosave and autocontinue systems for the case that we are reading a file.

Parameters:	current (str) – The currently tested element. last (str) – The last element of the list. status (str) – The status of the currently tested element.

_file_list_to_test_filtering()[source]¶: Unify the way we work before testing file contents.

classmethod _format_domain(extracted_domain)[source]¶

Format the extracted domain before passing it to the system.

Parameters:	extracted_domain (str) – The extracted domain.
Returns:	The formatted domain or IP to test.
Return type:	str

classmethod _print_header()[source]¶: Decide if we print or not the header.

classmethod bypass()[source]¶: Exit the script if [PyFunceble skip] is matched into the latest commit message.

classmethod colorify_logo(home=False)[source]¶

Print the colored logo based on global results.

Parameters:	home (bool) – Tell us if we have to print the initial coloration.

domain(domain=None, last_domain=None)[source]¶

Manage the case that we want to test only a domain.

Parameters:	domain (str) – The domain or IP to test. last_domain (str) – The last domain to test if we are testing a file. return_status (bool) – Tell us if we need to return the status.

file()[source]¶: Manage the case that need to test each domain of a given file path.

Note

1 domain per line.

file_url()[source]¶: Manage the case that we have to test a file

Note

1 URL per line.

classmethod reset_counters()[source]¶: Reset the counters when needed.

classmethod switch(variable, custom=False)[source]¶

Switch PyFunceble.CONFIGURATION variables to their opposite.

Parameters:	variable (str\|bool) – The variable name to switch. The variable should be an index our configuration system. If we want to switch a bool variable, we should parse it here. custom (bool) – Let us know if have to switch the parsed variable instead of our configuration index.
Returns:	The opposite of the configuration index or the given variable.
Return type:	bool
Raises:	`Exception` When the configuration is not valid. In other words, if the PyFunceble.CONFIGURATION[variable_name] is not a bool.

test(complete=False)[source]¶

Avoid confusion between self.domain which is called into __main__ and test() which should be called out of PyFunceble’s scope.

Parameters:	complete (bool) – Activate the return of a dictionnary with signigican data about the test.
Returns:	ACTIVE INACTIVE or INVALID.
Return type:	str\|list
Raises:	`Exception` When this method is called under `__name__ == '__main__'`

Note

This method should never be called in a __name__ == '__main__' context.

url(url_to_test=None, last_url=None)[source]¶

Manage the case that we want to test only a given url.

Parameters:	url_to_test (str) – The url to test. last_url (str) – The last url of the file we are testing (if exist)

Database¶

Problematics¶

How can we continuously test INACTIVE and INVALID domains or IP?
How can we reduce the number of whois requests over time?

Documentation¶

class PyFunceble.database.Inactive[source]¶

Logic behind the generation and the usage of a database system. The main idea behind this is to provide an inactive-db.json and test all inactive domain which are into to it regularly

_add_to_test(to_add)[source]¶

Add an element or a list of element into PyFunceble.CONFIGURATION['inactive_db'][self.file_path]['to_test'].

Parameters:	to_add (str\|list) – The domain, IP or URL to add.

_backup()[source]¶: Save the current database into the inactive-db.json file.

_merge()[source]¶: Merge the real database with the older one which has already been set into PyFunceble.CONFIGURATION["inactive_db"]

_reformat_historical_formating_error()[source]¶: Format the old format so it can be merged into the newer format.

_retrieve()[source]¶: Return the current content of the inactive-db.json file.

_timestamp()[source]¶

Get the timestamp where we are going to save our current list.

Returns:	The timestamp to append with the currently tested element.
Return type:	int\|str

add()[source]¶: Save the current :code.`PyFunceble.CONFIGURATION[‘to_test’]` into the current timestamp.

content()[source]¶

Get the content of the database.

Returns:	The content of the database.
Return type:	list

remove()[source]¶: Remove all occurence of PyFunceble.CONFIGURATION['to_test'] from the database.

to_test()[source]¶: Get the list to test for the next session.

class PyFunceble.database.Whois(expiration_date=None)[source]¶

Logic behind the whois database. Indeed, the idea is to implement #2.

Parameters:	expiration_date (str) – The extracted expiration date.

classmethod _authorization()[source]¶: Check if we are authorized to work with our database.

_backup()[source]¶: Backup the database into its file.

_retrieve()[source]¶: Retrieve the data from the database.

add()[source]¶: Add the currently tested element into the database.

get_expiration_date()[source]¶

Get the expiration date from the database.

Returns:	The expiration date from the database.
Return type:	str\|None

is_in_database()[source]¶: Check if the element is into the database.

is_time_older()[source]¶: Check if the current time is older than the one in the database.

Directory Structure¶

Problematic¶

How can we give make the output directory less annoying after an update?

Documentation¶

class PyFunceble.directory_structure.DirectoryStructure(production=False)[source]¶

Basically a backup/reconstructor of our output directory.

Parameters:	production (bool) – Tell the subsystem if we are preparing for production which imply the execution of the backup insteam of of the “reconstructore” mode.

classmethod _create_directory(directory, loop=False)[source]¶

Creates the given directory if it does not exists.

Parameters:	directory (str) – The directory to create. loop (bool) – Tell us if we are in the creation loop or not.

_get_structure()[source]¶

Get the structure we are going to work with.

Returns:	The structure we have to work with.
Return type:	dict

_restore_replace()[source]¶

Check if we need to replace “.gitignore” to “.keep”.

Returns:	The replacement status.
Return type:	bool

_update_structure_from_config(structure)[source]¶

Update the paths according to configs.

Parameters:	structure (dict) – The read structure.

backup()[source]¶

Backup the developer state of output/ in order to make it restorable: and portable for user.

delete_uneeded()[source]¶: Delete the directory which are not registered into our structure.

restore()[source]¶: Restore the ‘output/’ directory structure based on the dir_structure.json file.

Execution Time¶

Problematic¶

How to monitor the execution time of the session?

Documentation¶

class PyFunceble.execution_time.ExecutionTime(action='start', last=False)[source]¶

Set and return the exection time of the program.

Parameters:	action (str) – The action related the execution time. Can be start or stop. last (bool) – Tell the subsystem if we are at the very end of the file testing.

classmethod _authorization()[source]¶

Check the execution authorization.

Returns:	The authorization status.
Return type:	bool

classmethod _calculate(start=None, end=None)[source]¶

calculate the difference between starting and ending time.

Parameters:

start (int|str) – A starting time.
stop (int|str) – A ending time.

Returns:

A dict with following as index.

days

hours

minutes

seconds

as index.

Return type:

dict

_save(last=False)[source]¶

Save the current time to the file.

Parameters:	last (bool) – Tell us if we are at the very end of the file testing.

classmethod _starting_time()[source]¶: Set the starting time.

classmethod _stoping_time()[source]¶: Set the ending time.

format_execution_time(start=None, end=None)[source]¶

Format the calculated time into a human readable format.

Parameters:	start (int\|str) – A starting time. stop (int\|str) – A ending time.
Returns:	A human readable date.
Return type:	str

Expiration Date¶

Problematic¶

How can we get the expiration date of a given domain?

Documentation¶

class PyFunceble.expiration_date.ExpirationDate[source]¶

Get, format and return the expiration date of a domain, if exist.

_cases_management(regex_number, matched_result)[source]¶

A little internal helper of self.format. (Avoiding of nested loops)

Note

Please note that the second value of the case represent the groups in order [day,month,year].

This means that a [2,1,0] will be for example for a date in format 2017-01-02 where 01 is the month.

Parameters:	regex_number (int) – The identifiant of the regex. matched_result (list) – The matched result to format.
Returns:	A list representing the expiration date. The list can be “decoded” like `[day, month, year]`
Return type:	list\|None

classmethod _convert_1_to_2_digits(number)[source]¶

Convert 1 digit number to two digits.

Parameters:	number (str\|int) – A number or a digit string.
Returns:	A 2 or more digit string.
Return type:	str

classmethod _convert_or_shorten_month(data)[source]¶

Convert a given month into our unified format.

Parameters:	data (str) – The month to convert or shorten.
Returns:	The unified month name.
Return type:	str

_extract()[source]¶

Extract the expiration date from the whois record.

Returns:	The status of the domain.
Return type:	str

_format(date_to_convert=None)[source]¶

Format the expiration date into an unified format (01-jan-1970).

Parameters:	date_to_convert (str) – The date to convert. In other words, the extracted date.
Returns:	The formatted expiration date.
Return type:	str

get()[source]¶

Execute the logic behind the meaning of ExpirationDate + return the matched status.

Returns:	The status of the tested domain. Can be one of the official status.
Return type:	str

Generation¶

Problematic¶

How can we generate files which reflects the results of PyFunceble?

Documentation¶

class PyFunceble.generate.Generate(domain_status, source=None, expiration_date=None)[source]¶

Generate different sort of files.

Parameters:	domain_status (str) – The domain status. source (str) – The source of the given status. expiration_date (str) – The expiration date of the domain (if catched).

_analytic_host_file_directory()[source]¶: Return the analytic directory to write depending of the matched status.

classmethod _handle_non_existant_index()[source]¶: Handle and check that some configuration index exists.

_prints_status_file()[source]¶: Logic behind the printing (in file) when generating status file.

_prints_status_screen()[source]¶: Logic behind the printing (on screen) when generating status file.

_special_blogspot()[source]¶: Handle the blogspot SPECIAL case.

_special_wordpress_com()[source]¶: Handle the wordpress.com special case.

analytic_file(new_status, old_status)[source]¶

Generate Analytic/* files based on the given old and new statuses.

Parameters:	new_status (str) – The new status of the domain. old_status (str) – The old status of the domain.

down_status_file()[source]¶: Logic behind the down status when generating the status file.

info_files()[source]¶: Generate the hosts file, the plain list and the splitted lists.

invalid_status_file()[source]¶: Logic behind the invalid status when generating the status file.

status_file()[source]¶: Generate a file according to the domain status.

unified_file()[source]¶: Generate unified file. Understand by that that we use an unified table instead of a separate table for each status which could result into a misunderstanding.

up_status_file()[source]¶: Logic behind the up status when generating the status file.

valid_status_file()[source]¶: Logic behind the valis status when generating the status file.

HTTP Code¶

Problematic¶

How can we get the HTTP status code of the given domain or IP?

Documentation¶

class PyFunceble.http_code.HTTPCode[source]¶

Get and return the HTTP code status of a given domain.

_access()[source]¶

Get the HTTP code status.

Returns:	The matched HTTP status code.
Return type:	int\|None

get()[source]¶

Return the HTTP code status.

Returns:	The matched and formatted status code.
Return type:	str\|int\|None

IANA¶

Problematic¶

How can we get information from IANA?

Documentation¶

class PyFunceble.iana.IANA[source]¶

Logic behind the update of iana-domains-db.json

_extensions(block)[source]¶

Extract the extention from the given block. Plus get its referer.

Parameters:	block (str) – The line from the IANA database.

_referer(extension)[source]¶

Return the referer for the given extension.

Parameters:	extension (str) – A valid domain extension.
Returns:	The whois server to use to get the WHOIS record.
Return type:	str

update()[source]¶: Update the content of the iana-domains-db file.

Logs¶

Problematic¶

How can we efficiently generate and share logs?

Documentation¶

class PyFunceble.logs.Logs(output=None)[source]¶

Provide a clean and unique way to work with logs. Indeed, it’s not good to have logs spread around the code :smile:

Parameters:	output (str) – A path to the JSON file we are going to write.

classmethod _get_content(file)[source]¶

Get and return the content of the given log file.

Parameters:	file (str) – The file we have to get the content from.

:return The content of the given file. :rtype: dict

classmethod _write_content(content, file)[source]¶

Write the content into the given file.

Parameters:	content (dict) – The dict to write. file (str) – The file to write.

expiration_date(extracted)[source]¶

Logs the extracted expiration date.

Parameters:	extracted (str) – The extracted expiration date (from WHOIS record).

referer_not_found(extension)[source]¶

Logs the case that the referer was not found.

Parameters:	extension (str) – The extension of the domain we are testing.

whois(record)[source]¶

Logs the WHOIS record if needed.

Parameters:	record (str) – The record to log.

Lookup¶

Problematics¶

How can we get information from WHOIS records?
How can we check if a domain or IP have a valid pointer (nslookup)?

Documentation¶

class PyFunceble.lookup.Lookup[source]¶

Can be used to NSLOOKUP or WHOIS lookup.

classmethod nslookup()[source]¶: Implementation of UNIX nslookup.

classmethod whois(whois_server, domain=None, timeout=None)[source]¶

Implementation of UNIX whois.

Parameters:	whois_server (str) – The WHOIS server to use to get the record. domain (str) – The domain to get the whois record from. timeout (int) – The timeout to apply to the request.
Returns:	The whois record from the given whois server, if exist.
Return type:	str\|None

Mining¶

Problematic¶

How can we get the list of domain or URL which link to the desired domain, IPv4 or URL?

Documentation¶

class PyFunceble.mining.Mining[source]¶

Manage the minig subsystem.

_add(to_add)[source]¶

Add the currently mined information to the mined “database”.

Parameters:	to_add (dict) – The element to add.

_backup()[source]¶: Backup the mined informations.

_retrieve()[source]¶: Retrieve the mining informations.

list_of_mined()[source]¶

Provide the list of mined so they can be added to the list queue.

Returns:	The list of mined domains or URL.
Return type:	list

mine()[source]¶

Search for domain or URL related to the original URL or domain.

Returns:	The mined domains or URL.
Return type:	dict

process()[source]¶: Process the logic and structuration of the mining database.

remove()[source]¶: Remove the currently tested element from the mining data.

Percentage¶

Problematic¶

How can we calculate the percentage of each status?

Documentation¶

class PyFunceble.percentage.Percentage(domain_status=None, init=None)[source]¶

Calculation of the percentage of each status.

Parameters:	domain_status (str) – The status to increment. init (dict) – The data from a previous session we are continuing.

classmethod _calculate()[source]¶: Calculate the percentage of each status.

count()[source]¶: Count the number of domain for each status.

log()[source]¶: Print on screen and on file the percentages for each status.

Prints¶

Problematic¶

How can we print information on the screen and on file in a table format?

Documentation¶

class PyFunceble.prints.Prints(to_print, template, output_file=None, only_on_file=False)[source]¶

Print data on screen and into a file if needed. Template Possibilities: Percentage, Less, HTTP and any status you want.

Parameters:	to_print – The list of data to print. template (str) – The template to use. Note Available templates: `Percentage` `Less` `HTTP` any of the official status. output_file (str) – The path to the file to write. only_on_file (bool) – Tell us if we only have to print on file and not on screen.

_before_header()[source]¶: Print informations about PyFunceble and the date of generation of a file into a given path, if doesn’t exist.

_colorify(data)[source]¶

Retun colored string.

Parameters:	data (str) – The string to colorify.
Returns:	A colored string.
Return type:	str

_data_constructor(size)[source]¶

Construct the table of data according to given size.

Parameters:	size (list) – The maximal length of each string in the table.
Returns:	A dict with all information about the data and how to which what maximal size to print it.
Return type:	OrderedDict
Raises:	`Exception` If the data and the size does not have the same length.

classmethod _header_constructor(data_to_print, header_separator='-', column_separator=' ')[source]¶

Construct header of the table according to template.

Parameters:	data_to_print (list) – The list of data to print into the header of the table. header_separator (str) – The separator to use between the table header and our data. colomn_separator (str) – The separator to use between each colomns.
Returns:	The data to print in a list format.
Return type:	list

_json_print()[source]¶: Management of the json template.

classmethod _size_from_header(header)[source]¶

Get the size of each columns from the header.

Parameters:	header (dict) – The header template we have to get the size from.
Returns:	The maximal size of the each data to print.
Return type:	list

data()[source]¶

Management and input of data to the table.

Raises:	`Exception` When self.data_to_print is not a list.

header(do_not_print=False)[source]¶

Management and creation of templates of header. Please consider as “header” the title of each columns.

Parameters:	do_not_print (bool) – Tell us if we have to print the header or not.

Production¶

Problematic¶

How can we efficiently prepare the repository for push/production?

Documentation¶

class PyFunceble.production.Production(extern=False)[source]¶

Manage and provide the production preparation logic.

Parameters:	extern (bool) – Tell us if we do not have to execute the logic automatically. This allow method to be called.

classmethod is_dev_version()[source]¶: Check if the current branch is dev.

classmethod is_master_version()[source]¶: Check if the current branch is master.

Public Suffix¶

Problematic¶

How can we get the list of all possible or at least most used domain suffix?

Documentation¶

class PyFunceble.publicsuffix.PublicSuffix(live=True)[source]¶

Let us interact with the public suffix database.

classmethod _data()[source]¶: Get the database from the public suffix repository.

_extensions(line)[source]¶

Extract the extension from the given line.

Parameters:	line (str) – The line from the official public suffix repository.

load()[source]¶: Load the public suffix database into the system.

update()[source]¶: Update of the content of the public-suffix.json.

Referer¶

Problematic¶

How can we efficiently get the whois server to call for whois record request?

Documentation¶

class PyFunceble.referer.Referer[source]¶

Get the WHOIS server (referer) of the current domain extension according to the IANA database.

classmethod _iana_database()[source]¶

Convert iana-domains-db.json into a dictionnary.

Returns:	The content of the database in dictionnary format.
Return type:	dict

get()[source]¶: Return the referer aka the WHOIS server of the current domain extension.

Sort¶

Problematic¶

How can we format the list to test (and the outputted information) in a format other than the alphabetical format?

Documentation¶

class PyFunceble.sort.Sort[source]¶

Provide some sorting presets which we can parse to PyFunceble.helpers.List().custom_format().

classmethod hierarchical(element)[source]¶

The idea behind this method is to sort a list of domain hierarchicaly.

Parameters:	element (str) – The element we are currently reading.
Returns:	The formatted element.
Return type:	str

Note

For a domain like aaa.bbb.ccc.tdl.

A normal sorting is done in the following order:

aaa
bbb
ccc
tdl

This method allow the sorting to be done in the following order:

tdl
ccc
bbb
aaa

classmethod standard(element)[source]¶

Implement the standard and alphabetical sorting.

Parameters:	element (str) – The element we are currently reading.
Returns:	The formatted element.
Return type:	str

Status¶

Problematic¶

How can we efficiently manage the statuses in function of the test type?

Documentation¶

Normal testing¶

class PyFunceble.status.Status(matched_status, invalid_source='IANA')[source]¶

Hanle the research of domain status in case we don’t use WHOIS or in case that WHOIS record is not readable nor exploitable.

Parameters:	matched_result (str) – The previously catched status.

handle()[source]¶

Handle the lack of WHOIS. :smile_cat:

Returns:	The strus of the domain after generating the files desired by the user.
Return type:	str

URL testing¶

class PyFunceble.status.URLStatus(catched_status)[source]¶

Generate everything around the catched status when testing for URL.

Parameters:	catched_status (str) – THe catched status.

handle()[source]¶: Handle the backend of the given status.

Syntax checking¶

class PyFunceble.status.SyntaxStatus(catched_status)[source]¶

Generate everything around the catched status when testing for Syntax.

Parameters:	catched_status (str) – THe catched status.

handle()[source]¶: Handle the backend of the given status.

Syntax Checking¶

Problematic¶

How can we check for syntax directly from the CLI?

Documentation¶

class PyFunceble.syntax.Syntax[source]¶

Manage everything around the Syntax testing.

classmethod get()[source]¶

Execute the logic behind the Syntax handling.

Returns:	The syntax status.
Return type:	str

URL Testing¶

Problematic¶

How can we test full URL?

Documentation¶

class PyFunceble.url.URL[source]¶

Manage everything around the URL testing.

classmethod get()[source]¶

Execute the logic behind the URL handling.

Returns:	The status of the URL.
Return type:	str

Known issues¶

This is the list of issues which are or will not be fixed (yet?).

Under Travis CI the coloration may not be shown.

Frequently Asked Questions¶

Note

Don’t want to leave your machine running PyFunceble for days?

Request your list to be tested at Dead-Hosts!

How to speed up a bit the process/test?¶

For now, we only invite you to disable the usage/request of the WHOIS record.

This action will let the script speed up because we only use the equivalent of nslookup and the HTTP status code to determine the status.

Warning

We use/request the WHOIS record in order to avoid a specific false positive case. Indeed, if we disable the usage/request all domains which are still registered but not assigned to a specific IP will be flagged as INACTIVE.

It’s not a problem if you keep/use the database system because the domain will be retested over time. But please keep in mind that without the database system we do not guarantee an accurate result.

Contributing¶

I’m really glad you’re reading this because we need contributions to make this tool one of the best tool on the Internet!

Submitting changes¶

Before anything, please keep in mind the following. If one or more of those conditions are not filled. Your Pull Request to PyFunceble may not be merged.

The master branch is used only for releasing a new or stable version of the code. That’s why we require that all contributions/modifications must be done under the dev or a new branch.

In order to gain some time and also understand what you are working on, your pull requests submission as long as your commit descriptions should be clear and complete as much as possible. We do an exception to commit with minor changed but big changes should have a complete description. Please ensure to use the following method when committing a big change.

Steps before commit¶

Note

The following do not apply if you do not touch the PyFunceble nor the tests directory.

$ # We format our code.
$ black PyFunceble && black tests/*.py
$ # We lint our code. Please make sure to fix all reported issues.
$ pylint PyFunceble && pylint tests/*.py
$ # We check the tests coverage. Please ensure that at lease 95% of the code is covered.
$ coverage run setup.py test && coverage report -m
$ # Prepare our files, :code:`version.yaml` and code for pushing.
$ PyFunceble --production

The commit¶

$ # There paragraph is optional if your changes/commits are obvious.
$ git commit -S -m "A summary of the commit" -m "A paragraph
> or a sentence explaining what changed, why and its impact."

All your commits should be signed with PGP. (More information can be found on GitHub documentation)

Please note the usage of -S into the commit command which means that we sign the commit. The usage of PyFunceble --production update version.yaml and directory_structure_production.json automatically.

Coding conventions¶

We make sure that a method, a function, and a class have doctrings.
One line should not exceed 79 characters for docstring and 100 characters for long declaration/assignment. - Exception granted for regular expressions or long string assignment.
We use Black, The uncompromising Python code formatter, to format our code.
Our code should pass pylint PyFunceble && pylint tests/*.py with at least a score of 10.00/10.00
We do not forget to follow the steps before any commits.

Contributors¶

Thanks to those awesome peoples for their awesome and crazy idea(s), contribution(s) and or issue report which made or make PyFunceble a better tool.

 _______ _                 _          _                              _
|__   __| |               | |        | |                            | |
   | |  | |__   __ _ _ __ | | _____  | |_ ___    _   _  ___  _   _  | |
   | |  | '_ \ / _` | '_ \| |/ / __| | __/ _ \  | | | |/ _ \| | | | | |
   | |  | | | | (_| | | | |   <\__ \ | || (_) | | |_| | (_) | |_| | |_|
   |_|  |_| |_|\__,_|_| |_|_|\_\___/  \__\___/   \__, |\___/ \__,_| (_)
                                                  __/ |
                                                 |___/

dnmTX - @dnmTX
jawz101 - @jawz101
Mitchell Krog - @mitchellkrogza
Odyseus - @Odyseus
Reza Rizqullah - @ybreza
ScriptTiger - @ScriptTiger
speedmann - @speedmann
WaLLy3K - @WaLLy3K
xxcriticxx - @xxcriticxx

Special Thanks¶

Thanks to those awesome organization(s), tools and or people(s) for

Their awesome documentation
Their awesome repository
Their awesome tool/software/source code
Their breaking reports
Their contributions
Their current work/purpose
Their promotion of Py-Funceble
Their support
Their testings reports

which helped and/or still help me build, test and or make PyFunceble a better tool.

 _______ _                 _          _                              _
|__   __| |               | |        | |                            | |
   | |  | |__   __ _ _ __ | | _____  | |_ ___    _   _  ___  _   _  | |
   | |  | '_ \ / _` | '_ \| |/ / __| | __/ _ \  | | | |/ _ \| | | | | |
   | |  | | | | (_| | | | |   <\__ \ | || (_) | | |_| | (_) | |_| | |_|
   |_|  |_| |_|\__,_|_| |_|_|\_\___/  \__\___/   \__, |\___/ \__,_| (_)
                                                  __/ |
                                                 |___/

Adam Warner - @PromoFaux
Adblock Plus - @adblockplus
Mitchell Krog - @mitchellkrogza
Mohammad Fares - @faressoft
Pi-Hole - @pi-hole
Public Suffix List - @publicsuffix
Reza Rizqullah - @ybreza
ScriptTiger - @ScriptTiger
SMed79 - @SMed79
IANA - ICANN
yEd Graph Editor - yWorks

Supporting the project¶

PyFunceble, Dead-Hosts, and all other analog projects are powered by free time and a lot of coffee!

This project helps you and/or you like it?

They use PyFunceble!¶

Just a list of tool/repository/code/infrastructure which uses or used PyFunceble!

License¶

MIT License

Copyright (c) 2017-2018 Nissar Chababy

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

Code of Conduct¶

Our Pledge¶

In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.

Our Standards¶

Examples of behavior that contributes to creating a positive environment include:

Using welcoming and inclusive language
Being respectful of differing viewpoints and experiences
Gracefully accepting constructive criticism
Focusing on what is best for the community
Showing empathy towards other community members

Examples of unacceptable behavior by participants include:

The use of sexualized language or imagery and unwelcome sexual attention or advances
Trolling, insulting/derogatory comments, and personal or political attacks
Public or private harassment
Publishing others’ private information, such as a physical or electronic address, without explicit permission
Other conduct which could reasonably be considered inappropriate in a professional setting

Our Responsibilities¶

Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.

Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.

Scope¶

This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.

Enforcement¶

Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at contact@funilrys.com. The project team will review and investigate all complaints, and will respond in a way that it deems appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.

Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project’s leadership.

Attribution¶

This Code of Conduct is adapted from the Contributor Covenant, version 1.4, available at http://contributor-covenant.org/version/1/4.

Discovered any issue?¶

Want to make it public? Then fill a new issue.

Want to talk with @funilrys about it? Then stay in touch with Fun Ilrys on Keybase (preferred) or send an email to contactTATATAfunilrysTODTOTODcom.

Don’t want to fill an issue on GitHub? You can still inform us about it on our issues team on Keybase.

Need help?¶

Do you need help with the usage or application of PyFunceble? We can help!

Want to talk with @funilrys about it? Then stay in touch with Fun Ilrys on Keybase (preferred) or send an email to contactTATATAfunilrysTODTOTODcom.

Otherwise, let’s talk about it on our support team or our main team on Keybase!

Event	Shared	URL
No WHOIS server (referer) is found.	The extension of the currently tested domain.	`https://pyfunceble.funilrys.com/api/no-referer`
The expiration date is not correctly formatted.	The extracted expiration date. The currently tested domain. The currently used WHOIS server (DNS) name.	`https://pyfunceble.funilrys.com/api/date-format`

The tool to check the availability or syntax of domains, IPv4 or URL.¶

What can PyFunceble do?¶

PyFunceble in action¶

Domain testing¶

URL testing¶

Domain and URL testing with simple output¶

File content testing¶

File content testing with IDNA/Punycode conversion¶

Generated files/outputs¶

Dead-Hosts as place to use PyFunceble!¶

Why Dead-Hosts?¶

How do Dead-Hosts work?¶

History of Dead-Hosts¶

Installation¶

Requirements¶

Python 3.6+¶

colorama¶

domain2idna¶

PyYAML¶

requests¶

setuptools¶

urllib3¶

Get PyFunceble¶

Stable version¶

Using pip¶

From PyPi¶

From GitHub¶

Using the AUR (for Arch Linux users)¶

With makepkg¶

With your favorite AUR helper¶

Pure Python method¶

Development version¶

For development¶

For usage¶

Using pip¶

Using the AUR (for Arch Linux users)¶

Pure Python method¶

First steps¶

Update¶

Stable version¶

Using pip¶

From PyPi¶

From GitHub¶

Using the AUR (for Arch Linux users)¶

With makepkg¶

With your favorite AUR helper¶

Pure Python method¶

Development version¶

For development¶

For usage¶

Using pip¶

Using the AUR (for Arch Linux users)¶

Pure Python method¶

Configuration¶

Location¶

Problematics¶

Clone¶

Travis CI¶

Linux and MacOS (Darwin Kernel)¶

Windows¶

Custom location¶

Autoconfiguration¶

adblock¶

auto_continue¶

command¶

command_before_end¶

custom_ip¶

days_between_db_retest¶

debug¶

filter¶

generate_hosts¶

generate_json¶

header_printed¶

hierarchical_sorting¶

iana_whois_server¶

idna_conversion¶

inactive_database¶

less¶

local¶

logs¶

`colorama`¶

`domain2idna`¶

`PyYAML`¶

`requests`¶

`setuptools`¶

`urllib3`¶

Using `pip`¶

Using `pip`¶

Using `pip`¶

Using `pip`¶

`adblock`¶

`auto_continue`¶

`command`¶

`command_before_end`¶

`custom_ip`¶

`days_between_db_retest`¶

`debug`¶

`filter`¶

`generate_hosts`¶

`generate_json`¶

`header_printed`¶

`hierarchical_sorting`¶

`iana_whois_server`¶

`idna_conversion`¶

`inactive_database`¶

`less`¶

`local`¶

`logs`¶

`mining`¶

`no_files`¶

`no_whois`¶

`plain_list_domain`¶

`quiet`¶

`referer`¶

`seconds_before_http_timeout`¶

`share_logs`¶

`show_execution_time`¶

`show_percentage`¶

`simple`¶

`split`¶

`syntax`¶

`travis`¶

`travis_autosave_commit`¶

`travis_autosave_final_commit`¶

`travis_autosave_minutes`¶

`travis_branch`¶

`unified`¶

`user_agent`¶

`verify_ssl_certificate`¶

`whois_database`¶

`outputs`¶

`outputs[default_files]`¶

`outputs[default_files][dir_structure]`¶

`outputs[default_files][iana]`¶

`outputs[default_files][inactive_db]`¶

`outputs[default_files][results]`¶

`outputs[default_files][public_suffix]`¶

`outputs[domains]`¶

`outputs[domains][directory]`¶

`outputs[domains][filename]`¶

`outputs[hosts]`¶

`outputs[hosts][directory]`¶

`outputs[hosts][filename]`¶

`outputs[json]`¶

`outputs[json][directory]`¶

`outputs[json][filename]`¶

`outputs[analytic]`¶

`outputs[analytic][directories]`¶

`outputs[analytic][directories][parent]`¶

`outputs[analytic][directories][potentially_down]`¶

`outputs[analytic][directories][potentially_up]`¶

`outputs[analytic][directories][up]`¶

`outputs[analytic][directories][suspicious]`¶

`outputs[analytic][filenames]`¶

`outputs[analytic][filenames][potentially_down]`¶

`outputs[analytic][filenames][potentially_up]`¶

`outputs[analytic][filenames][up]`¶

`outputs[analytic][filenames][suspicious]`¶

`outputs[logs]`¶

`outputs[logs][directories]`¶