The tool to check the availability or syntax of domains, IPv4 or URL.¶
PyFunceble is the little sister of Funceble which was archived on 13th March 2018.
Its main objective is to provide the availability of domains, IPs and since recently URL by generating an accurate result based on results from WHOIS, NSLOOKUP and HTTP status codes.
PyFunceble is currently running actively and daily with the help of Travis CI under 60+ repositories. It is used to clean or test the availability of data which are present in hosts files, list of IP, list of domains, block lists or even AdBlock filter lists.
PyFunceble provides some useful features for continuous testing.
As an example, its auto-continue system coupled with its auto-save system allows it to run nice and smoothly under Travis CI without even reaching Travis CI time restriction. In the other side, its internal inactive database system let INACTIVE
and INVALID
caught domains, IPs or URLs being automatically retested over time on next run.
What can PyFunceble do?¶
- Test of the availability of a domain.
- Test of the availability of an IPv4.
- Test of the availability of an URL.
- Test of the availability of a domain/DNS name in a private or local network.
- Test of the availability of an IPv4 in a private or local network.
- Test of the availability of an URL in a private or local network.
- Test of the syntax of a domain.
- Test of the syntax of an IPv4.
- Test of the syntax of an URL.
- Test of domain or IP which are present into an Adblock formatted file.
- Test from a given raw link.
- Save test result(s) on file(s) (hosts file, plain text and/or JSON format).
- Show test result(s) on screen.
- Show percentage of each status (
ACTIVE
,INACTIVE
,INVALID
) - Sort outputs hierarchically.
- “Mining” of domain or IP which are related to the tested element.
- Auto-continuation of tests in case of system crash or script stop.
- Filtering of a file content.
- This feature will let us for example test all blogspot domain of the given file no matter the content of the file.
- Set the user-agent to use for the tests.
- Give some analytic depending of the HTTP status code (
ACTIVE
,POTENTIALLY_ACTIVE
,POTENTIALLY_INACTIVE
,SUSPICIOUS
). - Retest overtime of
INACTIVE
andINVALID
domains. - Print the execution time on screen and file.
- Customization of the different option via command-line arguments or configuration file.
- Continuous tests under Travis CI with the help of an auto saving and database system.
- Set branch to push the result to for the autosaving system.
- Set the minimal time before we autosave in order to avoid Travis CI limitation.
- Set a command to execute at the end of the test.
- Set the commit message for the autosaving system.
- … and a lot more!
PyFunceble in action¶
Domain testing¶
URL testing¶
Domain and URL testing with simple output¶
File content testing¶
File content testing with IDNA/Punycode conversion¶
Generated files/outputs¶
Dead-Hosts as place to use PyFunceble!¶
Why Dead-Hosts?¶
We are conscient that PyFunceble may run for days, that’s why we offer you to request your list to be tested at Dead-Hosts.
How do Dead-Hosts work?¶
Dead-Hosts is like a wrapper of PyFunceble. Indeed, we use Travis CI containers in order to test lists with PyFunceble.
Once a list is set up inside our infrastructure, PyFunceble will
test the list regularly and the Dead-Hosts infrastructure will produce a
clean.list
file which represents the list of domains/IP/URL
which remains or became ACTIVE
.
In addition, if needed, we can set up a list of GitHub username to @ping once a new test is finished.
History of Dead-Hosts¶
The project started on 3rd March 2017 at funilrys/dead-hosts.
The original idea was to test Funceble against hosts file in order to find bugs inside Funceble but also letting me have a fewer long hosts file.
On 23rd January 2018, I (funilrys) decided to shut funilrys/dead-hosts down. Indeed, as it became impossible to test all members of the project without having to wait weeks, I decided that it was time to move to another level.
That was the beginning of Dead-Hosts.
Today Dead-Hosts’s objective is to provide to project/list maintainers or individuals - with the help of PyFunceble - more information about their favorite project/list or domains, IP or URL.
Installation¶
Requirements¶
Here is the list of requirements:
- Python 3.6+
colorama
domain2idna
PyYAML
requests
setuptools
urllib3
Python 3.6+¶
As we use for example
print('hello', end=' ')
which does not exist in Python 2.x and as I wanted to give a priority to Python 3, Python 3 is required.
colorama
¶
As we use some coloration coloration, colorama
is required.
domain2idna
¶
As we propose the conversion of domains to IDNA, domain2idna
is required.
Note
domain2idna
is maintained and developed by Nissar Chababy (@funilrys), the main developer of PyFunceble.
Its source code can be found on GitHub.
PyYAML
¶
As our configuration file is written in .yaml
, PyYAML
is required.
requests
¶
As we use requests
when calling all Lookup()
methods, requests
is required.
setuptools
¶
As we use install_requires=xx
inside our setup.py
, setuptools
is required.
urllib3
¶
You should normally already have it. But as we handle some of its errors while using requests
, urllib3
is required.
Get PyFunceble¶
Stable version¶
Using pip
¶
Choose your repository, install and enjoy PyFunceble!
From PyPi¶
$ pip3 install PyFunceble
From GitHub¶
$ pip3 install git+https://github.com/funilrys/PyFunceble.git@master#egg=PyFunceble
Using the AUR (for Arch Linux users)¶
The package can be found at https://aur.archlinux.org/packages/python-pyfunceble/.
With makepkg¶
$ wget https://aur.archlinux.org/cgit/aur.git/plain/PKGBUILD?h=python-pyfunceble
$ makepkg
$ sudo pacman -U python-pyfunceble*.tar.xz
With your favorite AUR helper¶
Warning
We do not recommend any AUR helper but keep in mind that some AUR helpers are “better” than other. For more information about your current (or any other) AUR helper please report to the ArchWiki page.
$ yourFavoriteAurHelper -S python-pyfunceble
Pure Python method¶
Execute the following and enjoy PyFunceble!
$ git clone https://github.com/funilrys/PyFunceble.git
$ cd PyFunceble
$ python3 setup.py test && python3 setup.py install
Development version¶
The development version of PyFunceble represents the dev
branch.
It’s intended for the development of next features but is always at a usable state.
Indeed, We should not push to the dev
branch until we are sure that the new commit does not break or introduce critical issue under PyFunceble.
For development¶
Execute the following and let’s hack PyFunceble!
Note
We highly recommend you to develop PyFunceble under a virtualenv
.
$ git clone https://github.com/funilrys/PyFunceble.git
$ cd PyFunceble && git checkout dev && virtualenv venv
$ source venv/bin/activate && pip3 install -e .
Note
After installing with:
$ source venv/bin/activate && pip3 install -e .
- you only need to update the repository.
- you don’t have to rerun the
pip
command.
For usage¶
Using pip
¶
Execute one of the following and enjoy PyFunceble!
From PyPi
$ pip3 install PyFunceble-dev
From GitHub
$ pip3 install git+https://github.com/funilrys/PyFunceble.git@dev#egg=PyFunceble
Using the AUR (for Arch Linux users)¶
The package can be found at https://aur.archlinux.org/packages/python-pyfunceble-dev/.
With makepkg
$ wget https://aur.archlinux.org/cgit/aur.git/plain/PKGBUILD?h=python-pyfunceble-dev
$ makepkg
$ sudo pacman -U python-pyfunceble-dev*.tar.xz
With your favorite AUR helper
Warning
We do not recommend any AUR helper but keep in mind that some AUR helpers are “better” than other. For more information about your current (or any other) AUR helper please report to the ArchWiki page.
$ yourFavoriteAurHelper -S python-pyfunceble-dev
Pure Python method¶
Execute the following and enjoy PyFunceble!
$ git clone https://github.com/funilrys/PyFunceble.git
$ cd PyFunceble && git checkout dev
$ python3 setup.py test && python3 setup.py install
Update¶
Stable version¶
Using pip
¶
Choose your repository, install and enjoy PyFunceble!
From PyPi¶
$ pip3 install --upgrade PyFunceble
From GitHub¶
$ pip3 install --upgrade git+https://github.com/funilrys/PyFunceble.git@master#egg=PyFunceble
Using the AUR (for Arch Linux users)¶
With makepkg¶
$ wget https://aur.archlinux.org/cgit/aur.git/plain/PKGBUILD?h=python-pyfunceble
$ makepkg
$ sudo pacman -U python-pyfunceble*.tar.xz
With your favorite AUR helper¶
Warning
We do not recommend any AUR helper but keep in mind that some AUR helpers are “better” than other. For more information about your current (or any other) AUR helper please report to the ArchWiki page.
$ yourFavoriteAurHelper -Syu python-pyfunceble
Pure Python method¶
Execute the following and enjoy PyFunceble!
$ cd PyFunceble
$ git checkout master && git fetch origin && git merge origin/master
$ python3 setup.py test && python3 setup.py install
Development version¶
For development¶
$ cd PyFunceble && git checkout dev
$ git fetch origin && git merge origin/dev
Note
As you previously installed with
$ . venv/bin/activate && pip3 install -e .
Only code/repository update is required.
For usage¶
Using pip
¶
Execute one of the following and enjoy PyFunceble!
From PyPi
$ pip3 install --upgrade PyFunceble-dev
From GitHub
$ pip3 install --upgrade git+https://github.com/funilrys/PyFunceble.git@dev#egg=PyFunceble
Using the AUR (for Arch Linux users)¶
With makepkg
$ wget https://aur.archlinux.org/cgit/aur.git/plain/PKGBUILD?h=python-pyfunceble-dev
$ makepkg
$ sudo pacman -U python-pyfunceble-dev*.tar.xz
With your favorite AUR helper
Warning
We do not recommend any AUR helper but keep in mind that some AUR helpers are “better” than other. For more information about your current (or any other) AUR helper please report to the ArchWiki page.
$ yourFavoriteAurHelper -Syu python-pyfunceble-dev
Pure Python method¶
Execute the following and enjoy PyFunceble!
$ cd PyFunceble && git checkout dev
$ git fetch origin && git merge origin/dev
$ python3 setup.py test && python3 setup.py install
Configuration¶
This page will try to detail each configuration available into .PyFunceble.yaml
along with the location of where we are looking for the configuration file.
Location¶
Problematics¶
- How can we create a more efficient way to work with configuration?
- How can we make the configuration file(s) available globally so that PyFunceble can be run everywhere in the user workspace?
To answer those problematics, we moved the configuration location elsewhere in the place where most users expect to have their configuration file(s).
Clone¶
If you cloned the repository and you’re trying to test from a cloned directory (the one with for example CONTRIBUTING.md
) we consider the configuration directory as the current one.
Note
This behavior allows us to not modify the way we develop PyFunceble.
Travis CI¶
Under Travis CI, we search or initiate the configuration at the directory we are currently located.
Warning
We don’t care about the distribution, as long as the TRAVIS_BUILD_DIR
environment variable is set, we search or initiate the configuration in the current directory.
Note
If you want to force the directory where we should work, you can initiate the PYFUNCEBLE_OUTPUT_DIR
environment variable with the path where we should work.
Linux and MacOS (Darwin Kernel)¶
Under Linux and MacOS, we look for the following directories in their order. If any configuration directory is found, the system proposes you to install them automatically on the first configuration file.
~/.config/PyFunceble
~/.PyFunceble
${PWD}
Note
If the parent directory does not exist, we move to the next possible location in the given order.
This means that under most Linux distributions and MacOS versions, we consider ~/.config/PyFunceble
as the configuration location.
But if the ~/.config
directory does not exist, we fallback to ~/.PyFunceble
as the configuration location.
Windows¶
As mentioned by Pat Altimore’s Blog, we used the Per user configuration files synchronized across domain joined machines via Active Directory Roaming
section in order to understand what we should do to find our configuration directory.
Under Windows, we look for the following directories in their order. If any configuration directory is found, the system proposes you to install them automatically on the first configuration file.
%APPDATA%\PyFunceble
(environnement variable)%CD%
Note
%CD%
is explained by the set command (set /?
):
%CD% - expands to the current directory string.
Note
If the parent directory does not exist, we move to the next possible location in the given order.
This means that under most Windows versions, we consider %APPDATA%\PyFunceble
- also know as C:\Users\userName\AppData\Roaming\PyFunceble
- as the configuration location.
But if the %APPDATA%
directory does not exist, we fall back to the current directory as the configuration location.
Custom location¶
Sometimes, you may find yourself in a position where you absolutely do not want PyFunceble to use its default configuration location.
For that reason, if you set your desired configuration location along with the PYFUNCEBLE_OUTPUT_DIR
environment variable, we take that location as the (default) configuration location.
Autoconfiguration¶
Sometimes, you may find yourself in a position that you do not or you can’t answer the question which asks you if you would like to install the default configuration file.
For that reason, if you set PYFUNCEBLE_AUTO_CONFIGURATION
as an environment variable with what you want an assignment, we do not ask that question. We simply do what we have to do without asking anything.
adblock
¶
Type:
boolean
Default value:
False
Description: Enable / disable the adblock format decoding.
Note
If this index is set to True
, every time we read a given file, we try to extract the elements that are present.
We basically only decode the adblock format.
Note
If this index is set to False
, every time we read a given file, we will consider one line as an element to test.
auto_continue
¶
Type:
boolean
Default value:
True
Description: Enable / disable the auto continue system.
command
¶
Type:
string
Default value:
""
Description: Set the command to run before each commit (except the final one).
Note
The parsed command is called only if auto_continue
and travis
are set to True
.
command_before_end
¶
Type:
string
Default value:
""
Description: Set the command to run before the final commit.
Note
The parsed command is called only if auto_continue
and travis
are set to True
.
Note
Understand by final commit the commit which will deliver the last element we have to test.
custom_ip
¶
Type:
string
Default value:
"0.0.0.0"
Description: Set the custom IP to use when we generate a line in the hosts file format.
Note
This index has no effect if generate_hosts
is set to False
.
days_between_db_retest
¶
Type:
integer
Default value:
1
Description: Set the number of day(s) between each retest of the
INACTIVE
andINVALID
elements which are present intoinactive_db.json
.
Note
This index has no effect if inactive_database
is set to False
.
debug
¶
Type:
boolean
Default value:
False
Description: Enable / disable the generation of debug file(s).
Note
This index has no effect if logs
is set to False
Warning
Do not touch this index unless you a have good reason to.
Warning
Do not touch this index unless you have been invited to.
filter
¶
Type:
string
Default value:
""
Description: Set the element to filter.
Note
This index should be initiated with a regular expression.
generate_hosts
¶
Type:
boolean
Default value:
True
Description: Enable / disable the generation of the hosts file(s).
generate_json
¶
Type:
boolean
Default value:
False
Description: Enable / disable the generation of the JSON file(s).
header_printed
¶
Type:
boolean
Default value:
False
Description: Say to the system if the header has been already printed or not.
Warning
Do not touch this index unless you have a good reason to.
hierarchical_sorting
¶
Type:
boolean
Default value:
False
Description: Say to the system if we have to sort the list and the outputs in a hierarchical order.
iana_whois_server
¶
Type:
string
Default value:
whois.iana.org
Description: Set the server to call to get the
whois
referer of a given element.
Note
This index is only used when generating the iana-domains-db.json
file.
Warning
Do not touch this index unless you a have good reason to.
idna_conversion
¶
Type:
boolean
Default value:
False
Description: Tell the system to convert all domains to IDNA before testing.
Note
We use domain2idna for the conversion.
inactive_database
¶
Type:
boolean
Default value:
True
Description: Enable / Disable the usage of a database to store the
INACTIVE
andINVALID
element to retest overtime.
less
¶
Type:
boolean
Default value:
True
Description: Enable / Disable the output of every information of screen.
local
¶
Type:
boolean
Default value:
False
Description: Enable / Disable the execution of the test(s) in a local or private network.
logs
¶
Type:
boolean
Default value:
True
Description: Enable / Disable the output of all logs.
mining
¶
Type:
boolean
Default value:
True
Description: Enable / Disable the mining subsystem.
no_files
¶
Type:
boolean
Default value:
False
Description: Enable / Disable the generation of any file(s).
no_whois
¶
Type:
boolean
Default value:
False
Description: Enable / Disable the usage of
whois
in the tests.
plain_list_domain
¶
Type:
boolean
Default value:
False
Description: Enable / Disable the generation of the plain list of elements sorted by statuses.
Warning
Do not touch this index unless you a have good reason to.
quiet
¶
Type:
boolean
Default value:
False
Description: Enable / Disable the generation of output on the screen.
referer
¶
Type:
string
Default value:
""
Description: Set the referer of the element that is currently under test.
Warning
Do not touch this index unless you a have good reason to.
seconds_before_http_timeout
¶
Type:
integer
Default value:
3
Description: Set the timeout to apply to every HTTP status code request.
Note
This index must be a multiple of 3
.
show_execution_time
¶
Type:
boolean
Default value:
False
Description: Enable / disable the output of the execution time.
show_percentage
¶
Type:
boolean
Default value:
True
Description: Enable / disable the output of the percentage of each status.
simple
¶
Type:
boolean
Default value:
False
Description: Enable / disable the simple output mode.
Note
If this index is set to True
, the system will only return the result inf format: tested.element STATUS
.
split
¶
Type:
boolean
Default value:
True
Description: Enable / disable the split of the results files.
Note
Understand with “results files” the mirror of what is shown on screen.
syntax
¶
Type:
boolean
Default value:
False
Description: Enable / disable the syntax (only) testing.
Warning
If this index is set to True
, we ONLY check for syntax, not availability.
travis
¶
Type:
boolean
Default value:
False
Description: Enable / disable the Travis CI autosaving system.
Warning
Do not activate this index unless you are using PyFunceble under Travis CI.
travis_autosave_commit
¶
Type:
string
Default value:
"PyFunceble - AutoSave"
Description: Set the default commit message we want to use when have to commit (save) but our tests are not yet completed.
travis_autosave_final_commit
¶
Type:
string
Default value:
"PyFunceble - Results"
Description: Set the default final commit message we want to use when we all tests are finished.
travis_autosave_minutes
¶
Type:
integer
Default value:
15
Description: Set the minimum of minutes we have to run before to automatically save our test results.
Note
As many services are setting a rate limit per IP, it’s a good idea to set this value between 1
and 15
minutes.
travis_branch
¶
Type:
string
Default value:
master
Description: Set the git branch where we are going to push our results.
unified
¶
Type:
boolean
Default value:
False
Description: Enable / Disable the generation of the unified results.
Note
This index has no effect if split
is set to True
.
user_agent
¶
Type:
string
Default value:
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"
Description: Set the User-Agent to use every time we are requesting something from a web server other than our API.
verify_ssl_certificate
¶
Type:
boolean
Default value:
False
Description: Enable / Disable the verification of the SSL/TLS certificate when testing for URL.
Warning
If you set this index to True
, you may get false positive result.
Indeed if the certificate is not registered to the CA or is simply invalid and the domain is still alive, you will always get INACTIVE
as output.
whois_database
¶
Type:
boolean
Default value:
True
Description: Enable / Disable the usage of the whois database to avoid/bypass whois server requests rate limit.
outputs
¶
Type:
dict
Description: Set the needed output tree/names.
Warning
If you choose to change anything please consider deleting our output/
directory and the dir_structure*.json
files.
outputs[default_files]
¶
Type:
dict
Description: Set the default name of some important files.
outputs[default_files][dir_structure]
¶
Type:
string
Default value:
dir_structure.json
Description: Set the default filename of the file which has the structure to re-construct.
Note
This index has no influence with dir_structure_production.json
outputs[default_files][iana]
¶
Type:
string
Default value:
iana-domains-db.json
Description: Set the default filename of the file which has the formatted copy of the IANA root zone database.
outputs[default_files][inactive_db]
¶
Type:
string
Default value:
inactive_db.json
Description: Set the default filename of the file which will save the list of elements to retest overtime.
outputs[default_files][results]
¶
Type:
string
Default value:
results.txt
Description: Set the default filename of the file which will save the formatted copy of the public suffix database.
outputs[default_files][public_suffix]
¶
Type:
string
Default value:
public-suffix.json
Description: Set the default filename of the file which will save the mirror of what is shown on screen.
outputs[domains]
¶
Type:
dict
Description: Set the default name of some important files related to the
plain_list_domain
index.
outputs[domains][directory]
¶
Type:
string
Default value:
domains/
Description: Set the default directory where we have to save the plain list of elements for each status.
outputs[domains][filename]
¶
Type:
string
Default value:
list
Description: Set the default filename of the file which will save the plain list of elements.
outputs[hosts]
¶
Type:dict
Description: Set the default name of some important files related to the
generate_hosts
index.
outputs[hosts][directory]
¶
Type:
string
Default value:
hosts/
Description: Set the default directory where we have to save the hosts files of the elements for each status.
outputs[hosts][filename]
¶
Type:
string
Default value:
hosts
Description: Set the default filename of the file which will save the hosts files of the elements.
outputs[json]
¶
Type:dict
Description: Set the default name of some important files related to the
generate_json
index.
outputs[json][directory]
¶
Type:
string
Default value:
json/
Description: Set the default directory where we have to save the JSON files of the elements for each status.
outputs[json][filename]
¶
Type:
string
Default value:
dump.json
Description: Set the default filename of the file which will save the JSON files of the elements.
outputs[analytic]
¶
Type:dict
Description: Set the default name of some important files and directories related to the
generate_hosts
index.
outputs[analytic][directories]
¶
Type:
dict
Description: Set the default name of some important directories related to the
http_codes[active]
index.
outputs[analytic][directories][parent]
¶
Type:
string
Default value:
Analytic/
Description: Set the default directory where we are going to put everything related to the HTTP analytic.
outputs[analytic][directories][potentially_down]
¶
Type:
string
Default value:
POTENTIALLY_INACTIVE/
Description: Set the default directory where we are going to put all potentially inactive data.
outputs[analytic][directories][potentially_up]
¶
Type:
string
Default value:
POTENTIALLY_INACTIVE/
Description: Set the default directory where we are going to put all potentially active data.
outputs[analytic][directories][up]
¶
Type:
string
Default value:
POTENTIALLY_INACTIVE/
Description: Set the default directory where we are going to put all active data.
outputs[analytic][directories][suspicious]
¶
Type:
string
Default value:
SUSPICIOUS/
Description: Set the default directory where we are going to put all suspicious data.
outputs[analytic][filenames]
¶
Type:
dict
Description: Set the default name of some important files related to the
http_codes[active]
index and the HTTP analytic subsystem.
outputs[analytic][filenames][potentially_down]
¶
Type:
string
Default value:
down_or_potentially_down
Description: Set the default filename where we are going to put all potentially inactive data.
outputs[analytic][filenames][potentially_up]
¶
Type:
string
Default value:
potentially_up
Description: Set the default filename where we are going to put all potentially active data.
outputs[analytic][filenames][up]
¶
Type:
string
Default value:
active_and_merged_in_results
Description: Set the default filename where we are going to put all active data.
outputs[analytic][filenames][suspicious]
¶
Type:
string
Default value:
suspicious_and_merged_in_results
Description: Set the default filename where we are going to put all suspicious data.
outputs[logs]
¶
Type:
dict
Description: Set the default name of some important files and directories related to the
logs
index.
outputs[logs][directories]
¶
Type:dict
Description: Set the default name of some important directories related to the
logs
index.
outputs[logs][directories][date_format]
¶
Type:
string
Default value:
date_format/
Description: Set the default directory where we are going to put everything related to the data when the dates are in the wrong format.
outputs[logs][directories][no_referer]
¶
Type:
string
Default value:
no_referer/
Description: Set the default directory where we are going to put everything related to the data when no referer is found.
outputs[logs][directories][parent]
¶
Type:
string
Default value:
no_referer/
Description: Set the default directory where we are going to put everything related to the data when no referer is found.
outputs[logs][directories][percentage]
¶
Type:
string
Default value:
percentage/
Description: Set the default directory where we are going to put everything related to percentages.
outputs[logs][directories][whois]
¶
Type:
string
Default value:
whois/
Description: Set the default directory where we are going to put everything related to whois data.
Note
This is the location of all files when the debug
index is set to True
.
outputs[logs][filenames]
¶
Type:
dict
Description: Set the default filenames of some important files related to the
logs
index.
outputs[logs][filenames][auto_continue]
¶
Type:
string
Default value:
continue.json
Description: Set the default filename where we are going to put the data related to the auto continue subsystem.
Note
This file is allocated if the auto_continue
is set to True
.
outputs[logs][filenames][execution_time]
¶
Type:
string
Default value:
execution.log
Description: Set the default filename where we are going to put the data related to the execution time.
Note
This file is allocated if the show_execution_time
is set to True
.
outputs[logs][filenames][percentage]
¶
Type:
string
Default value:
percentage.txt
Description: Set the default filename where we are going to put the data related to the percentage.
Note
This file is allocated if the show_percentage
is set to True
.
outputs[main]
¶
Type:
string
Default value:
""
Description: Set the default location where we have to generate the
parent_directory
directory and its dependencies.
outputs[parent_directory]
¶
Type:
string
Default value:
output/
Description: Set the directory name of the parent directory which will contain all previously nouned directories.
status
¶
Type:
dict
Description: Set the needed, accepted and status name.
status[list]
¶
Type:
dict
Description: Set the needed and accepted status name.
Warning
All status should be in lowercase.
status[list][valid]
¶
Type:
list
Default value:
["valid","syntax_valid","valid_syntax"]
Description: Set the accepted
VALID
status.
Note
This status is only shown if the syntax
index is activated.
status[list][up]
¶
Type:
list
Default value:
["up","active"]
Description: Set the accepted
ACTIVE
status.
status[list][generic]
¶
Type:
list
Default value:
["generic"]
Description: Set the accepted
generic
status.
Note
This status is the one used to say the system that we have to print the complete information on the screen.
status[list][http_active]
¶
Type:
list
Default value:
["http_active"]
Description: Set the accepted status for the
outputs[analytic][filenames][up]
index.
status[list][down]
¶
Type:
list
Default value:
["down","inactive", "error"]
Description: Set the accepted status
INACTIVE
index.
status[list][invalid]
¶
Type:
list
Default value:
["ouch","invalid"]
Description: Set the accepted status
INVALID
index.
status[list][potentially_down]
¶
Type:
list
Default value:
["potentially_down", "potentially_inactive"]
Description: Set the accepted status for the
outputs[analytic][filenames][potentially_down]
index.
status[list][potentially_up]
¶
Type:
list
Default value:
["potentially_up", "potentially_active"]
Description: Set the accepted status for the
outputs[analytic][filenames][potentially_up]
index.
status[list][suspicious]
¶
Type:
list
Default value:
["strange", "hum", "suspicious"]
Description: Set the accepted status for the
outputs[analytic][filenames][suspicious]
index.
status[official]
¶
Type:
dict
Description: Set the official status name.
Note
Those status are the ones that are printed on the screen.
Warning
After any changes here please delete dir_structure.json
and the output/
directory.
status[official][up]
¶
Type:
string
Default value:
ACTIVE
Description: Set the returned status for the
ACTIVE
case.
status[official][down]
¶
Type:
string
Default value:
INACTIVE
Description: Set the returned status for the
INACTIVE
case.
status[official][invalid]
¶
Type:
string
Default value:
INVALID
Description: Set the returned status for the
INVALID
case.
status[official][valid]
¶
Type:
string
Default value:
VALID
Description: Set the returned status for the
VALID
case.
Note
This status is only shown if the syntax
index is activated.
http_codes
¶
Type:
dict
Description: Handle the interpretation of each status codes when we do and generate our analytic data.
http_codes[active]
¶
Type:
boolean
Default value:
True
Description: Enable / Disable the usage of the HTTP status code extraction.
http_codes[list]
¶
Type:
dict
Description: Categorize the http status code as mentioned in the documentation related to the
HTTP Code
column.
http_codes[list][up]
¶
Type:
list
- Default value:
- 100 - 101 - 200 - 201 - 202 - 203 - 204 - 205 - 206Description: List the HTTP status codes which are considered as
ACTIVE
.
http_codes[list][potentially_down]
¶
Type:
list
- Default value:
- 400 - 402 - 403 - 404 - 409 - 410 - 412 - 414 - 415 - 416Description: List the HTTP status code which are considered as
INACTIVE
orPOTENTIALLY_INACTIVE
.
http_codes[list][potentially_up]
¶
Type:
list
- Default value:
- 000 - 300 - 301 - 302 - 303 - 304 - 305 - 307 - 403 - 405 - 406 - 407 - 408 - 411 - 413 - 417 - 500 - 501 - 502 - 503 - 504 - 505Description: List the HTTP status code which are considered as
ACTIVE
orPOTENTIALLY_ACTIVE
.
links
¶
Type:
dict
Description: Set the list of links which can be used/called by the system when needed.
Note
The objective of this index is to avoid hardcoded links when the configuration file is readable.
links[api_date_format]
¶
Type:
string
Default value:
https://pyfunceble.funilrys.com/api/date-format
Description: Set the link to use when we share logs.
links[api_no_referer]
¶
Type:
string
Default value:
https://pyfunceble.funilrys.com/api/no-referer
Description: Set the link to use when we share logs.
links[config]
¶
Type:
string
Default value:
https://raw.githubusercontent.com/funilrys/PyFunceble/master/.PyFunceble_production.yaml
Description: Set the upstream link to the configuration file.
links[dir_structure]
¶
Type:
string
Default value:
https://raw.githubusercontent.com/funilrys/PyFunceble/master/dir_structure_production.json
Description: Set the upstream link to the directory structure dump file.
links[iana]
¶
Type:
string
Default value:
https://raw.githubusercontent.com/funilrys/PyFunceble/master/iana-domains-db.json
Description: Set the upstream link to the IANA zone file configuration file.
links[repo]
¶
Type:
string
Default value:
https://github.com/funilrys/PyFunceble
Description: Set the upstream link to the repository.
links[requirements]
¶
Type:
string
Default value:
https://raw.githubusercontent.com/funilrys/PyFunceble/master/requirements.txt
Description: Set the upstream link to the
requirements.txt
file.
links[psl]
¶
Type:
string
Default value:
https://raw.githubusercontent.com/funilrys/PyFunceble/master/public-suffix.json
Description: Set the upstream link to the public suffix database file.
counter
¶
Type:
dict
Description: Setup the internal counter.
Warning
The following is not intended for modification. Exception for debugging or special cases which requires an initiated counter.
counter[number]
¶
Type:
dict
Description: Setup the internal counter.
Warning
The following is not intended for modification. Exception for debugging or special cases which requires an initiated counter.
counter[number][down]
¶
Type:
integer
Default value:
0
Description: Setup the internal down counter.
Warning
The following is not intended for modification. Exception for debugging or special cases which requires an initiated counter.
counter[number][invalid]
¶
Type:
integer
Default value:
0
Description: Setup the internal invalid counter.
Warning
The following is not intended for modification. Exception for debugging or special cases which requires an initiated counter.
counter[number][tested]
¶
Type:
integer
Default value:
0
Description: Setup the internal tested counter.
Warning
The following is not intended for modification. Exception for debugging or special cases which requires an initiated counter.
counter[number][up]
¶
Type:
integer
Default value:
0
Description: Setup the internal up counter.
Warning
The following is not intended for modification. Exception for debugging or special cases which requires an initiated counter.
counter[percentage]
¶
Type:
dict
Description: Setup the internal percentage counter.
Warning
The following is not intended for modification. Exception for debugging or special cases which requires an initiated counter.
counter[percentage][down]
¶
Type:
integer
Default value:
0
Description: Setup the internal down percentage counter.
Warning
The following is not intended for modification. Exception for debugging or special cases which requires an initiated counter.
counter[percentage][invalid]
¶
Type:
integer
Default value:
0
Description: Setup the internal invalid percentage counter.
Warning
The following is not intended for modification. Exception for debugging or special cases which requires an initiated counter.
counter[percentage][up]
¶
Type:
integer
Default value:
0
Description: Setup the internal up percentage counter.
Warning
The following is not intended for modification. Exception for debugging or special cases which requires an initiated counter.
Usage¶
From a terminal¶
Detailed¶
Note
False
stand for deactivated when True
stand for activated.
-ad
| --adblock
¶
Switch the decoding of the adblock format.
Default value:
False
If this argument is activated the system will extract all domains or IP from the given adblock file.
-a
| --all
¶
Output all available information on the screen.
Default value:
False
When activated:
Domain Status Expiration Date Source HTTP Code
---------------------------------------------------------------------------------------------------- ----------- ----------------- ---------- ----------
pyfunceble.readthedocs.io ACTIVE Unknown NSLOOKUP 302
When deactivated:
Domain Status HTTP Code
---------------------------------------------------------------------------------------------------- ----------- ----------
pyfunceble.readthedocs.io ACTIVE 302
-c
| --auto-continue
| --continue
¶
Switch the value of the auto continue mode.
Default value:
True
This argument activates or deactivates the auto-continue subsystem. Indeed, as we can automatically continue if the script has been stopped, this switch allows us to disable or enable the usage of that specific subsystem.
--clean
¶
Clean all files under output.
As it is sometimes needed to clean our output/
directory, this argument does the job automatically.
Warning
This argument delete everything which are .keep
or .gitignore
--clean-all
¶
Clean all files under output and all file generated by PyFunceble.
Warning
This really deletes everything we generated without any warning.
--cmd
“something”¶
Pass a command before each commit (except the final one).
Default value:
''
Note
In this example, something
should be a script or a program which have to be executed when we reached the end of the given file.
Note
This argument is only used if --travis
or travis : true
(under .PyFunceble.yaml
) are activated.
--cmd-before-end "something"
¶
Pass a command before the results (final) commit under the Travis mode.
Default value:
''
Note
In this example, something
should be a script or a program which have to be executed when we reached the end of the given file.
Note
This argument is only used if --travis
or travis : true
(under .PyFunceble.yaml
) are activated.
--commit-autosave-message "something"
¶
Replace the default autosave commit message.
Default value:
PyFunceble - AutoSave
This argument allows us to set a custom commit message which is going to be used as commit message when saving.
Note
This argument is only used if --travis
or travis : true
(under .PyFunceble.yaml
) are used.
Note
This argument is only used if we have to split the work into multiple processes because a list is too long or the timeout is reached.
Warning
Please avoid the usage of [ci skip]
here.
--commit-results-message "something"
¶
Replace the default results (final) commit message.
Default value:
PyFunceble - Results
Note
This argument is only used if --travis
or travis : true
(under .PyFunceble.yaml
) are used.
Note
This argument is only used if we reached the end of the list we are or have to test.
-d "something"
| --domain "something"
¶
Set and test the given domain.
This argument will test and give the results of the tests of the given domain.
Note
For this argument (and only for this argument), we are converting the given string to lowercase.
-db
| --database
¶
Switch the value of the usage of a database to store inactive domains of the currently tested list.
Default value:
True
This argument will disable or enable the usage of a database which saves all INACTIVE and INVALID domain of the given file over time.
Note
The database is retested every x day(s), where x is the number set in -dbr "something"
.
-dbr "something"
¶
Set the numbers of days between each retest of domains present into the database of INACTIVE and INVALID domains.
Default value:
1
Note
This argument is only used if -db
or inactive_database : true
(under .PyFunceble.yaml
) are activated.
--debug
¶
Switch the value of the debug mode.
Default value:
False
This argument activates the debug mode. Under the debug mode, everything caught by the whois subsystem is saved.
Warning
Do not use this argument unless you have been told to.
--directory-structure
¶
Generate the directory and files that are needed and which does not exist in the current directory.
Want to start without anything? This argument generates the output directory automatically for you!
Note
In case of a file or directory not found issue, it’s recommended to remove the dir_structure.json
along with the output/ directory before using this argument.
-ex
| --execution
¶
Switch the default value of the execution time showing.
Default value:
False
Want to know the execution time of your test? Well, this argument will let you know!
-f "something"
| --file "something"
¶
Read the given file and test all domains inside it. If a URL is given we download and test the content of the given URL.
Note
We consider one line as one domain or one commented line. A line can be commented at the end.
Note
You can give a raw link and the system will download and test its content.
--filter "something"
¶
Domain to filter (regex).
Want to test all blogspot
from your list? This argument allows you to do that!
Note
This argument should be a regex expression.
--help
¶
Show the help message and exit.
-h
| --host
¶
Switch the value of the generation of hosts file.
Default value:
True
This argument will let the system know if it has to generate the hosts file version of each status.
--hierarchical
¶
Switch the value of the hierarchical sorting of the tested file.
Default value:
True
This argument will let the system know if we have to sort the list and our output in hierarchical order.
--http
¶
Switch the value of the usage of HTTP code.
Default value:
True
You don’t want to take the result of the HTTP code execution in consideration? This argument allows you to disable that!
Note
If activated the subsystem will bypass the HTTP status code extraction logic-representation.rst
--iana
¶
Update/Generate iana-domains-db.json.
This argument generates or updates iana-domains-db.json.
--idna
¶
Switch the value of the IDNA conversion.
Default value:
False
This argument allows the conversion of the domains using domain2idna
Warning
This feature is not supported yet for the URL testing.
-ip "something"
¶
Change the IP to print with the hosts files.
Default value:
0.0.0.0
--json
¶
Switch the value of the generation of the JSON formatted list of domains.
Default value:
False
--less
¶
When activated:
Domain Status HTTP Code
---------------------------------------------------------------------------------------------------- ----------- ----------
pyfunceble.readthedocs.io ACTIVE 302
When deactivated:
Domain Status Expiration Date Source HTTP Code
---------------------------------------------------------------------------------------------------- ----------- ----------------- ---------- ----------
pyfunceble.readthedocs.io ACTIVE Unknown NSLOOKUP 302
--local
¶
Switch the value of the local network testing.
Default value:
False
Want to run a test over a local or private network? This argument will disable the limitation which does not apply to private networks.
--link "something"
¶
Download and test the given file.
Want to test a raw link? This argument will download and test the given raw link.
-m
| --mining
¶
Switch the value of the mining subsystem usage.
Default value:
False
Want to find domain or URL linked to a domain in your list? This argument will exactly do that.
-n
| --no-files
¶
Switch the value the production of output files.
Default value:
False
Want to disable the production of the outputted files? This argument is for you!
-nl
| --no-logs
¶
Switch the value of the production of logs files in the case we encounter some errors.
Default value:
False
Don’t want any logs to go out of PyFunceble? This argument disables every logs subsystem.
-nu
| --no-unified
¶
Switch the value of the production unified logs under the output directory.
Default value:
True
This argument disables the generation of result.txt.
-nw
| --no-whois
¶
Switch the value the usage of whois to test domain’s status.
Default value:
False
Don’t want to use or take in consideration the results from whois
? This argument allows you to disable it!
-p
| --percentage
¶
Switch the value of the percentage output mode.
Default value:
True
This argument will disable or enable the generation of the percentage of each status.
--plain
¶
Switch the value of the generation of the plain list of domains.
Default value:
False:
Want to get a list with all domain for each status? The activation of this argument does the work while testing!
--production
¶
Prepare the repository for production.
Warning
Do not use this argument unless you have been told to, you prepare a Pull Request or you want to distribute your modified version of PyFunceble.
-psl
| --public-suffix
¶
Update/Generate public-suffix.json.
This argument will generate or update public-suffix.json.
-q
| --quiet
¶
Run the script in quiet mode.
Default value:
False
You prefer to run a program silently? This argument is for you!
-s
| --simple
¶
Switch the value of the simple output mode.
Default value:
False
Want as less as possible data on screen? This argument returns as less as possible on screen!
--split
¶
Switch the value of the split of the generated output
Default value:
True
Want to get the logs (copy of what you see on screen) on different files? This argument is suited to you!
--syntax
¶
Switch the value of the syntax test mode.
Default value:
False
-t "something"
| --timeout "something"
¶
Switch the value of the timeout.
Default value:
3
This argument will set the default timeout to apply everywhere it is possible to set a timeout.
--travis
¶
Switch the value of the Travis mode.
Default value:
False
Want to use PyFunceble under Travis CI? This argument is suited for your need!
-url "something"
| --url "something"
¶
Analyze the given URL.
Want to test the availability or an URL? Enjoy this argument!
Note
When we test the availability of an URL, we check the HTTP status code of the given URL.
-uf "something"
| --url-file "something"
¶
Read and test the list of URL of the given file. If a URL is given we download and test the content of the given URL.
Note
We consider one line as one URL to test.
Note
You can give a raw link and the system will download and test its content.
-ua "something"
| --user-agent "something"
¶
Set the user-agent to use and set every time we interact with everything which is not our logs sharing system.
-v
| --version
¶
Show the version of PyFunceble and exit.
-vsc
| --verify-ssl-certificate
¶
Switch the value of the verification of the SSL/TLS certificate when testing for URL.
Default value:
False
Warning
If you activate the verification of the SSL/TLS certificate, you may get false positive results.
Indeed if the certificate is not registered to the CA or is simply invalid and the domain is still alive, you will always get
INACTIVE
as output.
-wdb
| --whois-database
¶
Switch the value of the usage of a database to store whois data in order to avoid whois servers rate limit.
Default value:
True
Global overview¶
usage: PyFunceble [-ad] [-a] [-c] [--autosave-minutes AUTOSAVE_MINUTES]
[--clean] [--clean-all] [--cmd CMD]
[--cmd-before-end CMD_BEFORE_END]
[--commit-autosave-message COMMIT_AUTOSAVE_MESSAGE]
[--commit-results-message COMMIT_RESULTS_MESSAGE]
[-d DOMAIN] [-db] [-dbr DAYS_BETWEEN_DB_RETEST] [--debug]
[--directory-structure] [-ex] [-f FILE] [--filter FILTER]
[--help] [--hierarchical] [-h] [--http] [--iana] [--idna]
[-ip IP] [--json] [--less] [--local] [--link LINK] [-m] [-n]
[-nl] [-nu] [-nw] [-p] [--plain] [--production] [-psl] [-q]
[--share-logs] [-s] [--split] [--syntax] [-t TIMEOUT]
[--travis] [--travis-branch TRAVIS_BRANCH] [-u URL]
[-uf URL_FILE] [-ua USER_AGENT] [-v] [-vsc] [-wdb]
optional arguments:
-ad, --adblock Switch the decoding of the adblock format.
Configured value: False
-a, --all Output all available informations on screen.
Configured value: True
-c, --auto-continue, --continue
Switch the value of the auto continue mode.
Configured value: True
--autosave-minutes AUTOSAVE_MINUTES
Update the minimum of minutes before we start
committing to upstream under Travis CI.
Configured value: 15
--clean Clean all files under output.
--clean-all Clean all files under output and all file generated by
PyFunceble.
--cmd CMD Pass a command to run before each commit (except the
final one) under the travis mode. Configured
value: ''
--cmd-before-end CMD_BEFORE_END
Pass a command to run before the results (final)
commit under the Travis mode. Configured
value: ''
--commit-autosave-message COMMIT_AUTOSAVE_MESSAGE
Replace the default autosave commit message.
Configured value: 'PyFunceble -
AutoSave'
--commit-results-message COMMIT_RESULTS_MESSAGE
Replace the default results (final) commit message.
Configured value: 'PyFunceble -
Results'
-d DOMAIN, --domain DOMAIN
Set and test the given domain.
-db, --database Switch the value of the usage of a database to store
inactive domains of the currently tested list.
Configured value: True
-dbr DAYS_BETWEEN_DB_RETEST, --days-between-db-retest DAYS_BETWEEN_DB_RETEST
Set the numbers of days between each retest of
domains present into inactive-db.json.
Configured value: 1
--debug Switch the value of the debug mode.
Configured value: False
--directory-structure
Generate the directory and files that are needed and
which does not exist in the current directory.
-ex, --execution Switch the default value of the execution time
showing. Configured value: False
-f FILE, --file FILE Read the given file and test all domains inside it. If
a URL is given we download and test the content of the
given URL.
--filter FILTER Domain to filter (regex).
--help Show this help message and exit.
--hierarchical Switch the value of the hierarchical sorting of tested
file. Configured value: True
-h, --host Switch the value of the generation of hosts file.
Configured value: True
--http Switch the value of the usage of HTTP code.
Configured value: True
--iana Update/Generate `iana-domains-db.json`.
--idna Switch the value of the IDNA conversion.
Configured value: False
-ip IP Change the IP to print in the hosts files with the given one.
Configured value: '0.0.0.0'
--json Switch the value of the generation of the json list of
domain. Configured value: False
--less Output less informations on screen.
Configured value: False
--local Switch the value of the local network testing.
Configured value: True
--link LINK Download and test the given file.
-m, --mining Switch the value of the mining subsystem usage.
Configured value: False
-n, --no-files Switch the value of the production of output files.
Configured value: False
-nl, --no-logs Switch the value of the production of logs files in
the case we encounter some errors. Configured
value: False
-nu, --no-unified Switch the value of the production unified logs under
the output directory. Configured value:
True
-nw, --no-whois Switch the value the usage of whois to test domain's
status. Configured value: False
-p, --percentage Switch the value of the percentage output mode.
Configured value: True
--plain Switch the value of the generation of the plain list
of domain. Configured value: False
--production Prepare the repository for production.
-psl, --public-suffix
Update/Generate `public-suffix.json`.
-q, --quiet Run the script in quiet mode. Configured
value: False
--share-logs Switch the value of the sharing of logs.
Configured value: True
-s, --simple Switch the value of the simple output mode.
Configured value: False
--split Switch the value of the split of the generated output
files. Configured value: True
--syntax Switch the value of the syntax test mode.
Configured value: False
-t TIMEOUT, --timeout TIMEOUT
Switch the value of the timeout. Configured
value: 3
--travis Switch the value of the Travis mode.
Configured value: False
--travis-branch TRAVIS_BRANCH
Switch the branch name where we are going to push.
Configured value: 'master'
-u URL, --url URL Analyze the given URL.
-uf URL_FILE, --url-file URL_FILE
Read and test the list of URL of the given file. If a
URL is given we download and test the content of the
given URL.
-ua USER_AGENT, --user-agent USER_AGENT
Set the user-agent to use and set every time we
interact with everything which is not our logs sharing
system.
-v, --version Show the version of PyFunceble and exit.
-vsc, --verify-ssl-certificate
Switch the value of the verification of the SSL/TLS
certificate when testing for URL. Configured
value: False
-wdb, --whois-database
Switch the value of the usage of a database to store
whois data in order to avoid whois servers rate limit.
Configured value: True
Crafted with ♥ by Nissar Chababy (Funilrys) with the
help of https://pyfunceble.rtfd.io/en/master/contributors.html &&
https://pyfunceble.rtfd.io/en/master/special-thanks.html
From a Python script or module¶
Before continuing reading this part, You should know that I consider that you can speak in Python. If it’s not the case, well, it’s the time to learn Python!
As PyFunceble is written in Python, it can be easily imported and used inside a script. This part will represent a basic example of usage.
Basic example¶
"""
This is a basic example which prints one of the official output of PyFunceble.
Note:
* Official output: ACTIVE, INACTIVE, INVALID
"""
from PyFunceble import test as PyFunceble
from PyFunceble import url_test as PyFuncebleURL
print(PyFunceble(domain='google.com'))
print(PyFuncebleURL(url='https://google.com'))
"""
This is a basic example which checks syntax.
"""
from PyFunceble import syntax_check as PyFuncebleDomainSyntax
from PyFunceble import url_syntax_check as PyFuncebleURLSyntax
from PyFunceble import ipv4_syntax_check as PyFuncebleIPv4Syntax
print("google.com", PyFuncebleDomainSyntax(domain="google.com"))
print("https://google.com", PyFuncebleURLSyntax(url="https://google.com"))
print("216.58.207.46", PyFuncebleIPv4Syntax(ip="216.58.207.46"))
print("forest-jump", PyFuncebleDomainSyntax(domain="forest-jump"))
print("https://forest-jump", PyFuncebleURLSyntax(url="https://forest-jump"))
print("257.58.207.46", PyFuncebleIPv4Syntax(ip="257.58.207.46"))
Loop example¶
This part is unnecessary but we wanted to document it!!
"""
This is a loop example which tests a list of domain and processes some action
according to one of the official output of PyFunceble.
Note:
* Official output: ACTIVE, INACTIVE, INVALID
* You should always use PyFunceble().test() as it's the method which is especially
suited for `__name__ != '__main__'` usage.
"""
from PyFunceble import test as PyFunceble
from PyFunceble import url_test as PyFuncebleURL
DOMAINS = ["twitter.com", "google.com", "github.com", "github.comcomcom", "funilrys.co"]
def domain_status(domain_or_ip):
"""
Check the status of the given domain name or IP.
Argument:
- domain_or_ip: str
The domain or IPv4 to test.
Returns: str
The status of the domain.
"""
return PyFunceble(domain_or_ip)
def url_status(url):
"""
Check the status of the given url.
Argument:
- url: str
The URL to test.
Returns: str
The status of the URL.
"""
return PyFuncebleURL(url)
for domain in DOMAINS:
print(
"%s is %s and %s is %s"
% (
domain,
domain_status(domain),
"http://" + domain,
url_status("http://" + domain),
)
)
Advanced example¶
PyFunceble now allow you to get the following information as a dictionary. The objective behind this feature is to let you know more about the element you are testing.
{
"tested": None, # The tested element.
"expiration_date": None, # The expiration_date of the element if found.
"domain_syntax_validation": None, # The domain syntax validation status.
"http_status_code": None, # The status code of the tested element.
"ip4_syntax_validation": None, # The IPv4 syntax validation status.
"nslookup": [], # A list of IP of the tested element.
"status": None, # The status matched by PyFunceble.
"url_syntax_validation": None, # The url syntax validation status.
"whois_server": None, # The whois server if found.
"whois_record": None, # The whois record if whois_server is found.
}
To get that information simply work with our interface like follow :)
"""
This is an advanced example which prints some information about the tested element.
Note:
* Official output: ACTIVE, INACTIVE, INVALID
"""
from PyFunceble import test as PyFunceble
from PyFunceble import url_test as PyFuncebleURL
domain_testing = PyFunceble(domain='google.com', complete=True)
url_testing = PyFuncebleURL(url='https://google.com', complete=True)
print(domain_testing['nslookup'])
print(domain_testing['domain_syntax_validation'])
print(domain_testing['domain'], domain_testing['status'])
print(url_testing['nslookup'])
print(url_testing['domain_syntax_validation'])
print(url_testing['domain'], domain_testing['status'])
From Travis CI¶
As we offer an argument named --travis
to activate the usage of PyFunceble in a Travis CI instance, we document here what you need to know!
Configuration¶
Note
This part only present a commented .travis.yml
so that you can understand where to start.
If you need more practical examples, feel free to report to one of Dead-Hosts repositories which use PyFunceble with Travis CI.
env:
global:
# The following is your encrypted GitHub API key.
# Indeed as we are going to push to the repository, this is needed.
- secure: QQdKFquFFojFT9XJ1XZp4EMoDTVoXFgqZq8XU+sCVf+pJQR6d/oKBp8rnSTCnZizWOQXUjGXUUxUpSG/dYGyBLjo3rH3rsn9ciZHVfubxbwK860w4sqibl4DvhCv2rdsFtvzXnhm4P9OL3i+krKdewh9fxpNyUU58qOgfnS7mK9FcFhb8z5ak2sxU2XRZedwm6Ro0oyVKs8kFkL4YaADfNyAHlGTfr9rVmE52WXQXQENktb9gFgR2A8ZnmLy0BCMZGkPDShJnjRDWD4DErtasLmLQvWpzOBwdbVJTY6U9KDRXVNdC9lp5E5Ba/dc0y36q6vjfgJR+QchetOtHgNbKYbLB8c26Di90OZCFJsxMNcl1Wct4qFPXkFGvjXrISW6pbdPL5Plto0Ig3iLiulhYOPVArysMIk9ymtSXP+WE7VWX01LQ1fEkIoSfeVZ2caTnCmTsoHVGRRe978CojKaT7yU45kb15hcyDrzptQ8EP2hfxeh5F7KtueQ6Rsb9LFDZMkMDKflZn6a+bRhESlmWWmYB9stzGzTurQA1E1bcSACJ8A8hG5nHBzZYJ2S+OY0PE7UdyOJ0JK0qe/67d+F9ocQdIoFpDDTdgIjHerQnD2wRg1aKPzLDb4jJTpqgr5ssPrqUAKl3st7gyaAZzCEADPDnIBDjOJS+mFWbx9DKgc=
# This is the Git name we have to set. (git config user.name)
- GIT_NAME: Travis CI
# This is the Git Email we have to set. (git config user.email)
- GIT_EMAIL: dead-hosts@funilrys.com
# This is the full slug of the repository we are working with.
- TRAVIS_REPO_SLUG: dead-hosts/repository-structure
# This is the branch we have to checkout and push to.
- GIT_BRANCH: master
# This is the language we use.
language: python
# This is the python version we are going to use for the tests.
# Note: you can add any 3.x version to the list.
python:
- "3.6"
# The following will tell Travis CI to ends as fast as possible.
matrix:
fast_finish: true
# Here we are setting what Travis CI have to cache.
cache:
# We are caching pip3 as we use it to install PyFunceble
- pip3
install:
# We install the development version of PyFunceble. If you prefer the stable version replace `pyfunceble-dev` with `pyfunceble`.
- pip3 install pyfunceble-dev
# Our tests start here.
script:
# Let's say we want our results and our PyFunceble infrastructure to be saved in a directory called `PyFunceble-tests`
# We move inside it.
- cd PyFunceble-tests
# We test the file `my_awesome_list` which is located inside the current directory.
# Note: we precise the `--travis` argument here,
# but you work without it if you set `travis: true` inside your `.PyFunceble.yaml`
- PyFunceble --travis -f my_awesome_list --plain
# The following initiate email notification logic.
notifications:
# As we want to get a mail on failure and on status change, we set the following.
on_success: change
on_failure: always
Getting a GitHub token¶
For the secure
index of the .travis.yml
file, you have to generate a new GitHub token.
After you got your token, please write it or save it in a safe place as you’re going to need it every time you’re going to interact with Travis CI.
Note
The scope to set is public_repo
but you can also set others depending on your needs.
Encrypting the token for Travis CI usage¶
To encrypt the token simply replace and execute the following according to your personal case.
$ travis encrypt 'GH_TOKEN=theGeneratedToken' -r 'The content of TRAVIS_REPO_SLUG' --add
Warning
Please do not execute the following explicitly without replacing theGeneratedToken
with your previously generated GitHub token and The content of TRAVIS_REPO_SLUG
with your repository slug.
Note
The usage of --add
ensure that the travis
program automatically add the secure
index to the .travis.yml
file.
Columns¶
Because PyFunceble provides multiple information in a table, we chose to document them all. The objective of this page is to reflect what the code actually do in a more clear and understandable way.
Domains¶
This column is one of the basic ones as it gives us the name of the last tested domain or IP.
Status¶
There’s 3 possible output for this column.
ACTIVE¶
This status is returned when one of the following cases is met:
- We can extract the expiration date from
Lookup().whois()
.- Please note that we don’t check if the date is in the past.
Lookup().nslookup()
don’t return an error.- Please note that we don’t read the returned value.
HTTPCode().get()
return one the following code[100, 101, 200, 201, 202, 203, 204, 205, 206]
.
INACTIVE¶
This status is returned when all the following cases are met:
- We can’t extract the expiration date from
Lookup().whois()
. Lookup().nslookup()
don’t return an error.
INVALID¶
This status is returned when all the following cases are met:
Domain/IP does not match pass our syntax checker.
Domain extension is unregistered in IANA Root Zone Database.
Note
Understand by this that the extension is not present in the
iana-domains-db.json
file.
VALID¶
This status is returned when we are checking for syntax. It is the equivalent of ACTIVE
but for syntax checking.
Expiration Date¶
There’s two possible output for this column.
Unknown¶
Unknown
is returned when we could not extract the expiration date from Lookup().whois()
outputs.
A date¶
Only if we could extract the expiration date from Lookup().whois()
, the date becomes formatted like 02-jan-2017
.
Source¶
At this time, there’s 5 possible output for this column.
HTTP Code¶
This source is returned when all the following cases are met:
- We can’t extract the expiration date from
Lookup().whois()
. - The
INACTIVE
status is the one returned by other methods. HTTPCode().get()
outputs is one the following[100, 101, 200, 201, 202, 203, 204, 205, 206]
.
IANA¶
This source is always returned when the domain has the status INVALID
.
The usage of this source comes from the comparison of the element extension against the IANA Root Zone Database.
SYNTAX¶
This source is always returned when the domain has the status INVALID
or in the case that we are only checking for syntax instead of availability.
The usage of this source comes from the comparison of the element against our domain, IP or URL syntax validation system.
NSLOOKUP¶
This source is always returned when the taken decision of the status of the domain/IP comes from Lookup().nslookup()
outputs.
SPECIAL¶
As PyFunceble
grows, I thought that a bit of filtering for special cases would be great.
So I introduced the SPECIAL source.
Note
Please consider all 3 digits number that are listed in this section as the HTTP status code catched by HTTPCode().get()
.
*.blogspot.*
¶
- All
404
are returned asINACTIVE
- All
301
which are blocked by Google or does not exist are returned asINACTIVE
- All
302
which are blocked by Google are returned asINACTIVE
*.canalblog.com
¶
- All
404
are returned asINACTIVE
*.doubleclick.net
¶
- All
404
are returned asINACTIVE
*.liveadvert.com
¶
- All
404
are returned asINACTIVE
*.skyrock.com
¶
- All
404
are returned asINACTIVE
*.tumblr.com
¶
- All
404
are returned asINACTIVE
*.wordpress.com
¶
- All
301
which matchdoesn’t exist
are returned asINACTIVE
IP with range¶
- All IPv4 with a range (for example
0.0.0.0/24
) are returned asACTIVE
HTTP Code¶
Note
This section reflects exactly what the code does. So any changes made here should also be reported to the code or at least the configuration file.
Note
A ***
in this column means that it was impossible to catch the HTTP status code from the web server.
We have categorized the HTTP Code into 3 parts.
- Active
- Consider those ones like the one that influences HTTP source logic.
- Please note that the domain is automatically introduced into the official outputs but we keep a record of it
output/Analytic/ACTIVE
.
- Potentially Active
- If the domain status returned by other methods are different from
INACTIVE
orINVALID
and the HTTP status code is into that list, we save the domain intooutput/Analytic/POTENTIALLY_ACTIVE
- If the domain status returned by other methods are different from
- Inactive or potentially inactive
- If the domain status returned by other methods are different from
ACTIVE
and the HTTP status code is on that list, we save the domain intooutput/Analytic/POTENTIALLY_INACTIVE
- If the domain status returned by other methods are different from
As active¶
- 100 - Continue
- 101 - Switching Protocols
- 200 - OK
- 201 - Created
- 202 - Accepted
- 203 - Non-Authoritative Information
- 204 - No Content
- 205 - Reset Content
- 206 - Partial Content
As potentially active¶
- 000
- 300 - Multiple Choices
- 301 - Moved Permanently
- 302 - Found
- 303 - See Other
- 304 - Not Modified
- 305 - Use Proxy
- 307 - Temporary Redirect
- 403 - Forbidden
- 405 - Method Not Allowed
- 406 - Not Acceptable
- 407 - Proxy Authentication Required
- 408 - Request Timeout
- 411 - Length Required
- 413 - Request Entity Too Large
- 417 - Expectation Failed
- 500 - Internal Server Error
- 501 - Not Implemented
- 502 - Bad Gateway
- 503 - Service Unavailable
- 504 - Gateway Timeout
- 505 - HTTP Version Not Supported
As inactive or potentially inactive¶
- 400 - Bad Request
- 401 - Unauthorized
- 402 - Payment Required (Not in use but may be seen in the future)
- 404 - Not Found
- 409 - Conflict
- 410 - Gone
- 412 - Precondition Failed
- 414 - Request-URI Too Long
- 415 - Unsupported Media Type
- 416 - Requested Range Not Satisfiable
Logs Sharing¶
Why logs sharing?¶
We chose to initiate the logs sharing as some actions can really be random when working with millions of domains.
The idea and purpose of this feature are ONLY to make PyFunceble a better tool.
Logic representation¶
Don’t have time to review the code and want to know exactly what PyFunceble does in general? This flowchart is there for you!
Note
This is a simplified version of what we actually do.
Note
The diagrams were generated with the help of yEd Graph Editor.
Domain testing¶
IP testing¶
URL testing¶
API documentation¶
Note
This section document what you can call directly when you use PyFunceble as an imported module.
The tool to check the availability or syntax of domains, IPv4 or URL.
██████╗ ██╗ ██╗███████╗██╗ ██╗███╗ ██╗ ██████╗███████╗██████╗ ██╗ ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║ ██║████╗ ██║██╔════╝██╔════╝██╔══██╗██║ ██╔════╝
██████╔╝ ╚████╔╝ █████╗ ██║ ██║██╔██╗ ██║██║ █████╗ ██████╔╝██║ █████╗
██╔═══╝ ╚██╔╝ ██╔══╝ ██║ ██║██║╚██╗██║██║ ██╔══╝ ██╔══██╗██║ ██╔══╝
██║ ██║ ██║ ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝ ╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝
This submodule is the main entry of PyFunceble.
- Author:
- Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
- Special thanks:
- https://pyfunceble.readthedocs.io/en/master/special-thanks.html
- Contributors:
- http://pyfunceble.readthedocs.io/en/master/special-thanks.html
- Project link:
- https://github.com/funilrys/PyFunceble
- Project documentation:
- https://pyfunceble.readthedocs.io/en/master/
- Project homepage:
- https://funilrys.github.io/PyFunceble/
License:
MIT License
Copyright (c) 2017-2018 Nissar Chababy
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
-
PyFunceble.
ipv4_syntax_check
(ip)[source]¶ Check the syntax of the given IPv4.
Parameters: ip (str) – The IPv4 to check the syntax for. Returns: The syntax validity. Return type: bool
-
PyFunceble.
load_config
(under_test=False)[source]¶ Load the configuration.
Parameters: under_test (bool) – Tell us if we only have to load the configuration file (True) or load the configuration file and initate the output directory if it does not exist (False).
-
PyFunceble.
syntax_check
(domain)[source]¶ Check the syntax of the given domain.
Parameters: domain (str) – The domain to check the syntax for. Returns: The syntax validity. Return type: bool
-
PyFunceble.
test
(domain, complete=False)[source]¶ Test the availability of the given domain or IP.
Parameters: - domain (str) – The domain or IP to test.
- complete (bool) – Activate the return of a dict with some significant data from the test.
Returns: The status or the informations of the domain.
Return type: str|dict
Note
This function abstract and simplify for the access to the core for the end-user.
-
PyFunceble.
url_syntax_check
(url)[source]¶ Check the syntax of the given URL.
Parameters: url (str) – The URL to check the syntax for. Returns: The syntax validity. Return type: bool
-
PyFunceble.
url_test
(url, complete=False)[source]¶ Test the availability of the given URL.
Parameters: - url (str) – The URL to test.
- complete (bool) – Activate the return of a dict with some significant data from the test.
Returns: The status or the informations of the URL.
Return type: str|dict
Note
This function abstract and simplify for the access to the core for the end-user.
Code Documentation¶
Note
This section will document every part (except the API section) of our code base.
Helpers¶
Problematic¶
How can we avoid writing the same thing every time?
Documentation¶
The tool to check the availability or syntax of domains, IPv4 or URL.
██████╗ ██╗ ██╗███████╗██╗ ██╗███╗ ██╗ ██████╗███████╗██████╗ ██╗ ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║ ██║████╗ ██║██╔════╝██╔════╝██╔══██╗██║ ██╔════╝
██████╔╝ ╚████╔╝ █████╗ ██║ ██║██╔██╗ ██║██║ █████╗ ██████╔╝██║ █████╗
██╔═══╝ ╚██╔╝ ██╔══╝ ██║ ██║██║╚██╗██║██║ ██╔══╝ ██╔══██╗██║ ██╔══╝
██║ ██║ ██║ ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝ ╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝
This submodule will provide the helpers.
- Author:
- Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
- Special thanks:
- https://pyfunceble.readthedocs.io/en/master/special-thanks.html
- Contributors:
- http://pyfunceble.readthedocs.io/en/master/special-thanks.html
- Project link:
- https://github.com/funilrys/PyFunceble
- Project documentation:
- https://pyfunceble.readthedocs.io/en/master/
- Project homepage:
- https://funilrys.github.io/PyFunceble/
License:
MIT License
Copyright (c) 2017-2018 Nissar Chababy
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
-
class
PyFunceble.helpers.
Command
(command)[source]¶ Shell command execution.
Parameters: command (str) – The command to execute
-
class
PyFunceble.helpers.
Dict
(main_dictionnary=None)[source]¶ Dictionary manipulations.
Parameters: main_dictionnary (dict) – The dict we are working with. -
classmethod
from_json
(data)[source]¶ Convert a JSON formatted string into a dictionary.
Parameters: data (str) – A JSON formatted string to convert to dict format. Returns: The dict representation of the JSON formatted string. Return type: dict
-
classmethod
from_yaml
(data)[source]¶ Convert a YAML formatted string into a dictionary.
Parameters: data (str) – A YAML formatted string to convert to dict format. Returns: The dict representation of the YAML formatted string. Return type: dict
-
merge
(to_merge, strict=True)[source]¶ Merge the content of to_merge into the given main dictionnary.
Parameters: - to_merge (dict) – The dictionnary to merge.
- strict (bool) –
Tell us if we have to strictly merge lists.
True
: We follow index :code`False`: We follow element (content)
Returns: The merged dict.
Return type: dict
-
remove_key
(key_to_remove)[source]¶ Remove a given key from a given dictionary.
Parameters: key_to_remove (list|str) – The key(s) to delete. Returns: The dict without the given key(s). Return type: dict|None
-
rename_key
(key_to_rename, strict=True)[source]¶ Rename the given keys from the given dictionary.
Parameters: - key_to_rename (dict) – The key(s) to rename.
Expected format:
{old:new}
- strict – Tell us if we have to rename the exact index or the index which looks like the given key(s)
Returns: The well formatted dict.
Return type: dict|None
- key_to_rename (dict) – The key(s) to rename.
Expected format:
-
classmethod
-
class
PyFunceble.helpers.
Directory
(directory)[source]¶ Directory manipulation.
Parameters: directory (str) – A path to the directory to manipulate.
-
class
PyFunceble.helpers.
Download
(link, destination=None, return_data=False, verify_certificate=True)[source]¶ Download or return the content of the given link.
Parameters: - link (str) – The link to download.
- destination (str) – The location where we should save the downloaded content.
- return_data (bool) – Tell us if we need to return the page content or write its content into the given destination.
- verify_certificate (bool) – Tell us if we need to verify the SSL/TLS certificate.
-
class
PyFunceble.helpers.
File
(file)[source]¶ File treatment/manipulations.
Parameters: file (str) – A path to the file to manipulate. -
copy
(destination)[source]¶ Copy the given file to the destination.
Parameters: destination (str) – The destination of the copy.
-
-
class
PyFunceble.helpers.
Hash
(file_path=None, algorithm='sha512', only_hash=False, data=None)[source]¶ Get and return the hash a file with the given algorithm.
Parameters: - file_path (str) – The path to the file we have to hash.
- algorithm (str) – The algorithm to use.
- only_hash – Tell us if we only have to return the desired algorithm instead of the dummy dict format.
Note
The original version can be found at https://git.io/vFQrK.
-
class
PyFunceble.helpers.
List
(main_list=None)[source]¶ List manipulation.
Parameters: main_list (list) – The list to manipulate. -
custom_format
(key_method, reverse=False)[source]¶ Return a well formatted list. With the key_method as a function/method to format the elements before sorting.
Parameters: - key_method (function|method) – A function or method to use to format the readed element before sorting.
- reverse (bool) – Tell us if we have to reverse the list.
Returns: A sorted list.
Return type: list
-
-
class
PyFunceble.helpers.
Regex
(data, regex, **args)[source]¶ A simple implementation ot the python.re package
Parameters: - data (str) – The data to check.
- regex (str) – The regex to match.
- group (int) – The group to return.
- rematch (bool) –
Allow to return the matched groups into a formatted list.
Note
This is an implementation of Bash
${BASH_REMATCH}
- replace_with (str) – The value to replace the matched regex with.
- occurences (int) – The number of occurence(s) to replace.
- return_type – Tell us if we have to return the matched data or simply check if we matched (True) or not (False)
Download()
¶
-
class
PyFunceble.helpers.
Download
(link, destination=None, return_data=False, verify_certificate=True)[source] Download or return the content of the given link.
Parameters: - link (str) – The link to download.
- destination (str) – The location where we should save the downloaded content.
- return_data (bool) – Tell us if we need to return the page content or write its content into the given destination.
- verify_certificate (bool) – Tell us if we need to verify the SSL/TLS certificate.
-
text
()[source] Download the given link and return or save its
requests.text
at the given destination.Return type: mixed
Raises: Exception
If the status code is not
200
.
Command()
¶
-
class
PyFunceble.helpers.
Command
(command)[source] Shell command execution.
Parameters: command (str) – The command to execute -
_decode_output
(to_decode)[source] Decode the output of a shell command in order to be readable.
Parameters: to_decode – Output of a command to decode. Type: bytes Returns: The decoded output. Return type: str
-
execute
()[source] Execute the given command.
Returns: The output of the command. Return type: str
-
Regex()
¶
-
class
PyFunceble.helpers.
Regex
(data, regex, **args)[source] A simple implementation ot the python.re package
Parameters: - data (str) – The data to check.
- regex (str) – The regex to match.
- group (int) – The group to return.
- rematch (bool) –
Allow to return the matched groups into a formatted list.
Note
This is an implementation of Bash
${BASH_REMATCH}
- replace_with (str) – The value to replace the matched regex with.
- occurences (int) – The number of occurence(s) to replace.
- return_type – Tell us if we have to return the matched data or simply check if we matched (True) or not (False)
-
match
()[source] Used to get exploitable result of re.search
Returns: The data of the match status. Return type: mixed
-
matching_list
()[source] Return a list of the string which match the given regex.
-
not_matching_list
()[source] Return a list of string which don’t match the given regex.
-
replace
()[source] Used to replace a matched string with another.
Returns: The data after replacement. Return type: str
Dict()
¶
-
class
PyFunceble.helpers.
Dict
(main_dictionnary=None)[source] Dictionary manipulations.
Parameters: main_dictionnary (dict) – The dict we are working with. -
classmethod
from_json
(data)[source] Convert a JSON formatted string into a dictionary.
Parameters: data (str) – A JSON formatted string to convert to dict format. Returns: The dict representation of the JSON formatted string. Return type: dict
-
classmethod
from_yaml
(data)[source] Convert a YAML formatted string into a dictionary.
Parameters: data (str) – A YAML formatted string to convert to dict format. Returns: The dict representation of the YAML formatted string. Return type: dict
-
merge
(to_merge, strict=True)[source] Merge the content of to_merge into the given main dictionnary.
Parameters: - to_merge (dict) – The dictionnary to merge.
- strict (bool) –
Tell us if we have to strictly merge lists.
True
: We follow index :code`False`: We follow element (content)
Returns: The merged dict.
Return type: dict
-
remove_key
(key_to_remove)[source] Remove a given key from a given dictionary.
Parameters: key_to_remove (list|str) – The key(s) to delete. Returns: The dict without the given key(s). Return type: dict|None
-
rename_key
(key_to_rename, strict=True)[source] Rename the given keys from the given dictionary.
Parameters: - key_to_rename (dict) – The key(s) to rename.
Expected format:
{old:new}
- strict – Tell us if we have to rename the exact index or the index which looks like the given key(s)
Returns: The well formatted dict.
Return type: dict|None
- key_to_rename (dict) – The key(s) to rename.
Expected format:
-
to_json
(destination)[source] Save a dictionnary into a JSON file.
Parameters: destination (str) – A path to a file where we’re going to write the converted dict into a JSON format.
-
to_yaml
(destination, flow_style=False)[source] Save a dictionnary into a YAML file.
Parameters: destination (str) – A path to a file where we’re going to write the converted dict into a JSON format.
-
classmethod
List()
¶
-
class
PyFunceble.helpers.
List
(main_list=None)[source] List manipulation.
Parameters: main_list (list) – The list to manipulate. -
custom_format
(key_method, reverse=False)[source] Return a well formatted list. With the key_method as a function/method to format the elements before sorting.
Parameters: - key_method (function|method) – A function or method to use to format the readed element before sorting.
- reverse (bool) – Tell us if we have to reverse the list.
Returns: A sorted list.
Return type: list
-
format
()[source] Return a well formatted list. Basicaly, it’s sort a list and remove duplicate.
Returns: A sorted, without duplicate, list. Return type: list
-
merge
(to_merge, strict=True)[source] Merge to_merge into the given main list.
Parameters: - to_merge (list) – The list to merge.
- strict (bool) – Tell us if we have to respect index (True) or not (False).
Returns: The merged list.
Return type: list
-
Directory()
¶
-
class
PyFunceble.helpers.
Directory
(directory)[source] Directory manipulation.
Parameters: directory (str) – A path to the directory to manipulate. -
fix_path
(splited_path=None)[source] Fix the path of the given path.
Parameters: splited_path (list) – A list to convert to the right path. Returns: The fixed path. Return type: str
-
File()
¶
-
class
PyFunceble.helpers.
File
(file)[source] File treatment/manipulations.
Parameters: file (str) – A path to the file to manipulate. -
copy
(destination)[source] Copy the given file to the destination.
Parameters: destination (str) – The destination of the copy.
-
delete
()[source] Delete a given file path.
-
read
()[source] Read a given file path and return its content.
Returns: The content of the given file path. Return type: str
-
write
(data_to_write, overwrite=False)[source] Write or append data into the given file path.
Parameters: - data_to_write (str) – The data to write.
- overwrite (bool) – Tell us if we have to overwrite the content of the file we are working with.
-
Hash()
¶
-
class
PyFunceble.helpers.
Hash
(file_path=None, algorithm='sha512', only_hash=False, data=None)[source] Get and return the hash a file with the given algorithm.
Parameters: - file_path (str) – The path to the file we have to hash.
- algorithm (str) – The algorithm to use.
- only_hash – Tell us if we only have to return the desired algorithm instead of the dummy dict format.
Note
The original version can be found at https://git.io/vFQrK.
-
_hash_data
(algo)[source] Get hash of the given data.
Parameters: algo (str) – The algorithm to use.
-
_hash_file
(algo)[source] Get the hash of the given file
Parameters: algo (str) – The algorithm to use. Returns: The hexdigest of the data. Return type: str
-
get
()[source] Return the hash of the given file
AdBlock¶
Problematic¶
How can we efficiently decode AdBlock filter list?
Documentation¶
-
class
PyFunceble.adblock.
AdBlock
(list_from_file, aggressive=False)[source]¶ Provide the adblock decoding logic.
Parameters: list_from_file (list) – The file in list format. -
classmethod
_extract_base
(element)[source]¶ Extract the base of the given element.
Parameters: element (str|list) – The element we are working with.
-
classmethod
_format_decoded
(to_format, result=None)[source]¶ Format the exctracted adblock line before passing it to the system.
Parameters: - to_format (str) – The extracted line from the file.
- result (list) – A list of the result of this method.
Returns: The list of domains or IP to test.
Return type: list
-
_handle_options
(options)[source]¶ Handle the data from the options.
Parameters: options (list) – The list of options from the rule. Returns: The list of domains to return globally. Return type: list
-
classmethod
Auto-save¶
Note
Only Travis CI is actually supported.
Travis CI problematic¶
How can we bypass the default Travis CI timeout of 45 minutes?
Check¶
Problematic¶
How can we efficiently check the format of IP, domains, and URL?
Documentation¶
-
class
PyFunceble.check.
Check
(element=None)[source]¶ Provide a place to check several things around URL, IP or domain.
Parameters: element (str) – The element (URL, IP or domain) to check. -
is_domain_valid
(domain=None, subdomain_check=False)[source]¶ Check if the given domain is a valid.
Parameters: - domain (str) – The domain to validate.
- subdomain_check (bool) – Activate the subdomain checking.
Returns: The validity of the sub-domain.
Return type: bool
-
is_ip_range
(ip_to_check=None)[source]¶ Check if the given IP is a valid IPv4.
Parameters: ip_to_check (str) – The IP to test. Returns: The validity of the IP. Return type: bool Note
We only test IPv4 because for now we only them for now.
-
is_ip_valid
(ip_to_check=None)[source]¶ Check if the given IP is a valid IPv4.
Parameters: ip_to_check (str) – The IP to test. Returns: The validity of the IP. Return type: bool Note
We only test IPv4 because for now we only them for now.
-
is_subdomain
(domain=None)[source]¶ Check if the given subdomain is a subdomain.
Parameters: domain (str) – The domain to validate. Returns: The validity of the subdomain. Return type: bool
-
is_url_valid
(url=None, return_base=False, return_formatted=False)[source]¶ Check if the given URL is valid.
Parameters: - url (str) – The url to validate.
- return_base – Allow us the return of the url base (if URL formatted correctly).
- return_formatted (bool) – Allow us to get the URL converted to IDNA if the conversion is activated.
Returns: The validity of the URL or its base.
Return type: bool|str
-
Cleaning¶
Problematic¶
How can we clean the output/
directory so we do not have a collision between old and new files?
Documentation¶
-
class
PyFunceble.clean.
Clean
(list_to_test, clean_all=False)[source]¶ Provide the cleaning logic(s).
Parameters: - list_to_test (list|None) – The list of domains we are testing.
- clean_all (bool) – Tell the subsystem if we need to clean all. Which include, of course, the output directory but also all other file(s) generated by our system.
Configuration¶
Problematics¶
- How can we avoid the usage of
tool.py
? - How can we make personalization more simple?
Documentation¶
Load()
¶
-
class
PyFunceble.config.
Load
(path_to_config)[source]¶ Help us load the configuration(s) file(s).
Parameters: path_to_config (str) – The possible path to the configuration to load. -
classmethod
_install_directory_structure_file
()[source]¶ Download the latest version of dir_structure_production.json.
-
classmethod
Version()
¶
-
class
PyFunceble.config.
Version
(used=False)[source]¶ Compare the local with the upstream version.
Parameters: used (bool) – True: Version is configured for simple usage. False: Version compare local with upstream. -
classmethod
check_versions
(local, upstream)[source]¶ Compare the given versions.
Parameters: - local (list) – The local version converted by split_versions().
- upstream (list) – The upstream version converted by split_versions().
Returns: - True: local < upstream
- None: local == upstream
- False: local > upstream
Return type: bool|None
-
classmethod
check_versions_literally
(local, upstream)[source]¶ Compare the given versions literally.
Parameters: - local (str) – The local version converted by split_versions().
- upstream (str) – The upstream version converted by split_versions().
Returns: - True: local == upstream
- False: local != upstream
Return type: bool
-
classmethod
is_cloned
()[source]¶ Let us know if we are currently in the cloned version of PyFunceble which implicitly mean that we are in developement mode.
-
classmethod
right_url_from_version
(url)[source]¶ Convert the GitHub URL to the right one depending of the branch or version we are working with.
Parameters: url (str) – The URL to convert. Returns: The converted URL. Return type: str
-
classmethod
split_versions
(version, return_non_digits=False)[source]¶ Convert the versions to a shorter one.
Parameters: - version (str) – The version to split.
- return_non_digits (bool) – Activate the return of the non-digits parts of the splitted version.
Returns: The splitted version name/numbers.
Return type: list
-
classmethod
Core¶
Documentation¶
-
class
PyFunceble.core.
Core
(**args)[source]¶ Main entry to PyFunceble. Brain of the program. Also known as “put everything together to make the system works”.
Parameters: - domain_or_ip_to_test (str) – A domain or IP to test.
- file_path (str) – A path to a file to read and test.
- url_to_test (str) – A URL to test.
- url_file (str) – A path to a file which contains URL to test.
- link_to_test (str) – A link to a file to download and test.
- modulo_test – If set to True, it will tell the system that we are working as an exported module.
- modulo_test – bool
-
classmethod
_entry_management_url_download
(passed)[source]¶ Check if the given information is a URL. If it is the case, it download and update the location of file to test.
Parameters: passed (str) – The url passed to the system. Returns: The state of the check. Return type: bool
-
classmethod
_extract_domain_from_file
()[source]¶ Extract all non commented lines from the file we are testing.
Returns: The elements to test. Return type: list
-
_file_decision
(current, last, status=None)[source]¶ Manage the database, autosave and autocontinue systems for the case that we are reading a file.
Parameters: - current (str) – The currently tested element.
- last (str) – The last element of the list.
- status (str) – The status of the currently tested element.
-
classmethod
_format_domain
(extracted_domain)[source]¶ Format the extracted domain before passing it to the system.
Parameters: extracted_domain (str) – The extracted domain. Returns: The formatted domain or IP to test. Return type: str
-
classmethod
bypass
()[source]¶ Exit the script if
[PyFunceble skip]
is matched into the latest commit message.
-
classmethod
colorify_logo
(home=False)[source]¶ Print the colored logo based on global results.
Parameters: home (bool) – Tell us if we have to print the initial coloration.
-
domain
(domain=None, last_domain=None)[source]¶ Manage the case that we want to test only a domain.
Parameters: - domain (str) – The domain or IP to test.
- last_domain (str) – The last domain to test if we are testing a file.
- return_status (bool) – Tell us if we need to return the status.
-
file
()[source]¶ Manage the case that need to test each domain of a given file path.
Note
1 domain per line.
-
classmethod
switch
(variable, custom=False)[source]¶ Switch PyFunceble.CONFIGURATION variables to their opposite.
Parameters: - variable (str|bool) – The variable name to switch. The variable should be an index our configuration system. If we want to switch a bool variable, we should parse it here.
- custom (bool) – Let us know if have to switch the parsed variable instead of our configuration index.
Returns: The opposite of the configuration index or the given variable.
Return type: bool
Raises: Exception
When the configuration is not valid. In other words, if the PyFunceble.CONFIGURATION[variable_name] is not a bool.
-
test
(complete=False)[source]¶ Avoid confusion between self.domain which is called into __main__ and test() which should be called out of PyFunceble’s scope.
Parameters: complete (bool) – Activate the return of a dictionnary with signigican data about the test.
Returns: ACTIVE INACTIVE or INVALID.
Return type: str|list
Raises: Exception
When this method is called under
__name__ == '__main__'
Note
This method should never be called in a
__name__ == '__main__'
context.
Database¶
Problematics¶
- How can we continuously test
INACTIVE
andINVALID
domains or IP? - How can we reduce the number of whois requests over time?
Documentation¶
-
class
PyFunceble.database.
Inactive
[source]¶ Logic behind the generation and the usage of a database system. The main idea behind this is to provide an inactive-db.json and test all inactive domain which are into to it regularly
-
_add_to_test
(to_add)[source]¶ Add an element or a list of element into
PyFunceble.CONFIGURATION['inactive_db'][self.file_path]['to_test']
.Parameters: to_add (str|list) – The domain, IP or URL to add.
-
_merge
()[source]¶ Merge the real database with the older one which has already been set into
PyFunceble.CONFIGURATION["inactive_db"]
-
_reformat_historical_formating_error
()[source]¶ Format the old format so it can be merged into the newer format.
-
_timestamp
()[source]¶ Get the timestamp where we are going to save our current list.
Returns: The timestamp to append with the currently tested element. Return type: int|str
-
add
()[source]¶ Save the current :code.`PyFunceble.CONFIGURATION[‘to_test’]` into the current timestamp.
-
-
class
PyFunceble.database.
Whois
(expiration_date=None)[source]¶ Logic behind the whois database. Indeed, the idea is to implement #2.
Parameters: expiration_date (str) – The extracted expiration date. Check if we are authorized to work with our database.
Directory Structure¶
Problematic¶
How can we give make the output directory less annoying after an update?
Documentation¶
-
class
PyFunceble.directory_structure.
DirectoryStructure
(production=False)[source]¶ Basically a backup/reconstructor of our
output
directory.Parameters: production (bool) – Tell the subsystem if we are preparing for production which imply the execution of the backup insteam of of the “reconstructore” mode. -
classmethod
_create_directory
(directory, loop=False)[source]¶ Creates the given directory if it does not exists.
Parameters: - directory (str) – The directory to create.
- loop (bool) – Tell us if we are in the creation loop or not.
-
_get_structure
()[source]¶ Get the structure we are going to work with.
Returns: The structure we have to work with. Return type: dict
-
_restore_replace
()[source]¶ Check if we need to replace “.gitignore” to “.keep”.
Returns: The replacement status. Return type: bool
-
_update_structure_from_config
(structure)[source]¶ Update the paths according to configs.
Parameters: structure (dict) – The read structure.
-
classmethod
Execution Time¶
Problematic¶
How to monitor the execution time of the session?
Documentation¶
-
class
PyFunceble.execution_time.
ExecutionTime
(action='start', last=False)[source]¶ Set and return the exection time of the program.
Parameters: - action (str) – The action related the execution time. Can be start or stop.
- last (bool) – Tell the subsystem if we are at the very end of the file testing.
Check the execution authorization.
Returns: The authorization status. Return type: bool
-
classmethod
_calculate
(start=None, end=None)[source]¶ calculate the difference between starting and ending time.
Parameters: - start (int|str) – A starting time.
- stop (int|str) – A ending time.
Returns: A dict with following as index.
days
hours
minutes
seconds
as index.
Return type: dict
Expiration Date¶
Problematic¶
How can we get the expiration date of a given domain?
Documentation¶
-
class
PyFunceble.expiration_date.
ExpirationDate
[source]¶ Get, format and return the expiration date of a domain, if exist.
-
_cases_management
(regex_number, matched_result)[source]¶ A little internal helper of self.format. (Avoiding of nested loops)
Note
Please note that the second value of the case represent the groups in order
[day,month,year]
.This means that a
[2,1,0]
will be for example for a date in format2017-01-02
where01
is the month.Parameters: - regex_number (int) – The identifiant of the regex.
- matched_result (list) – The matched result to format.
Returns: A list representing the expiration date. The list can be “decoded” like
[day, month, year]
Return type: list|None
-
classmethod
_convert_1_to_2_digits
(number)[source]¶ Convert 1 digit number to two digits.
Parameters: number (str|int) – A number or a digit string. Returns: A 2 or more digit string. Return type: str
-
classmethod
_convert_or_shorten_month
(data)[source]¶ Convert a given month into our unified format.
Parameters: data (str) – The month to convert or shorten. Returns: The unified month name. Return type: str
-
_extract
()[source]¶ Extract the expiration date from the whois record.
Returns: The status of the domain. Return type: str
-
Generation¶
Problematic¶
How can we generate files which reflects the results of PyFunceble?
Documentation¶
-
class
PyFunceble.generate.
Generate
(domain_status, source=None, expiration_date=None)[source]¶ Generate different sort of files.
Parameters: - domain_status (str) – The domain status.
- source (str) – The source of the given status.
- expiration_date (str) – The expiration date of the domain (if catched).
-
_analytic_host_file_directory
()[source]¶ Return the analytic directory to write depending of the matched status.
-
classmethod
_handle_non_existant_index
()[source]¶ Handle and check that some configuration index exists.
-
analytic_file
(new_status, old_status)[source]¶ Generate
Analytic/*
files based on the given old and new statuses.Parameters: - new_status (str) – The new status of the domain.
- old_status (str) – The old status of the domain.
HTTP Code¶
Problematic¶
How can we get the HTTP status code of the given domain or IP?
IANA¶
Problematic¶
How can we get information from IANA?
Documentation¶
-
class
PyFunceble.iana.
IANA
[source]¶ Logic behind the update and usage of iana-domains-db.json
-
_extensions
(block)[source]¶ Extract the extention from the given block. Plus get its referer.
Parameters: block (str) – The line from the IANA database.
-
Logs¶
Problematic¶
How can we efficiently generate and share logs?
Documentation¶
-
class
PyFunceble.logs.
Logs
(output=None)[source]¶ Provide a clean and unique way to work with logs. Indeed, it’s not good to have logs spread around the code :smile:
Parameters: output (str) – A path to the JSON file we are going to write. -
classmethod
_get_content
(file)[source]¶ Get and return the content of the given log file.
Parameters: file (str) – The file we have to get the content from. :return The content of the given file. :rtype: dict
-
classmethod
_write_content
(content, file)[source]¶ Write the content into the given file.
Parameters: - content (dict) – The dict to write.
- file (str) – The file to write.
-
expiration_date
(extracted)[source]¶ Logs the extracted expiration date.
Parameters: extracted (str) – The extracted expiration date (from WHOIS record).
-
classmethod
Lookup¶
Problematics¶
- How can we get information from WHOIS records?
- How can we check if a domain or IP have a valid pointer (nslookup)?
Documentation¶
-
class
PyFunceble.lookup.
Lookup
[source]¶ Can be used to NSLOOKUP or WHOIS lookup.
-
classmethod
whois
(whois_server, domain=None, timeout=None)[source]¶ Implementation of UNIX whois.
Parameters: - whois_server (str) – The WHOIS server to use to get the record.
- domain (str) – The domain to get the whois record from.
- timeout (int) – The timeout to apply to the request.
Returns: The whois record from the given whois server, if exist.
Return type: str|None
-
classmethod
Mining¶
Problematic¶
How can we get the list of domain or URL which link to the desired domain, IPv4 or URL?
Documentation¶
-
class
PyFunceble.mining.
Mining
[source]¶ Manage the minig subsystem.
-
_add
(to_add)[source]¶ Add the currently mined information to the mined “database”.
Parameters: to_add (dict) – The element to add.
-
list_of_mined
()[source]¶ Provide the list of mined so they can be added to the list queue.
Returns: The list of mined domains or URL. Return type: list
-
Percentage¶
Problematic¶
How can we calculate the percentage of each status?
Prints¶
Problematic¶
How can we print information on the screen and on file in a table format?
Documentation¶
-
class
PyFunceble.prints.
Prints
(to_print, template, output_file=None, only_on_file=False)[source]¶ Print data on screen and into a file if needed. Template Possibilities: Percentage, Less, HTTP and any status you want.
Parameters: - to_print – The list of data to print.
- template (str) –
The template to use.
Note
Available templates:
Percentage
Less
HTTP
- any of the official status.
- output_file (str) – The path to the file to write.
- only_on_file (bool) – Tell us if we only have to print on file and not on screen.
-
_before_header
()[source]¶ Print informations about PyFunceble and the date of generation of a file into a given path, if doesn’t exist.
-
_colorify
(data)[source]¶ Retun colored string.
Parameters: data (str) – The string to colorify. Returns: A colored string. Return type: str
-
_data_constructor
(size)[source]¶ Construct the table of data according to given size.
Parameters: size (list) – The maximal length of each string in the table.
Returns: A dict with all information about the data and how to which what maximal size to print it.
Return type: OrderedDict
Raises: Exception
If the data and the size does not have the same length.
-
classmethod
_header_constructor
(data_to_print, header_separator='-', column_separator=' ')[source]¶ Construct header of the table according to template.
Parameters: - data_to_print (list) – The list of data to print into the header of the table.
- header_separator (str) – The separator to use between the table header and our data.
- colomn_separator (str) – The separator to use between each colomns.
Returns: The data to print in a list format.
Return type: list
-
classmethod
_size_from_header
(header)[source]¶ Get the size of each columns from the header.
Parameters: header (dict) – The header template we have to get the size from. Returns: The maximal size of the each data to print. Return type: list
Production¶
Problematic¶
How can we efficiently prepare the repository for push/production?
Public Suffix¶
Problematic¶
How can we get the list of all possible or at least most used domain suffix?
Sort¶
Problematic¶
How can we format the list to test (and the outputted information) in a format other than the alphabetical format?
Documentation¶
-
class
PyFunceble.sort.
Sort
[source]¶ Provide some sorting presets which we can parse to PyFunceble.helpers.List().custom_format().
-
classmethod
hierarchical
(element)[source]¶ The idea behind this method is to sort a list of domain hierarchicaly.
Parameters: element (str) – The element we are currently reading. Returns: The formatted element. Return type: str Note
For a domain like
aaa.bbb.ccc.tdl
.- A normal sorting is done in the following order:
aaa
bbb
ccc
tdl
- This method allow the sorting to be done in the following order:
tdl
ccc
bbb
aaa
-
classmethod
Status¶
Problematic¶
How can we efficiently manage the statuses in function of the test type?
Documentation¶
Normal testing¶
URL testing¶
Known issues¶
This is the list of issues which are or will not be fixed (yet?).
- Under Travis CI the coloration may not be shown.
Frequently Asked Questions¶
Note
Don’t want to leave your machine running PyFunceble for days?
Request your list to be tested at Dead-Hosts!
How to speed up a bit the process/test?¶
For now, we only invite you to disable the usage/request of the WHOIS record.
This action will let the script speed up because we only use the equivalent of nslookup
and the HTTP status code to determine the status.
Warning
We use/request the WHOIS record in order to avoid a specific false positive case.
Indeed, if we disable the usage/request all domains which are still registered but not assigned to a specific IP will be flagged as INACTIVE
.
It’s not a problem if you keep/use the database system because the domain will be retested over time. But please keep in mind that without the database system we do not guarantee an accurate result.
Contributing¶
I’m really glad you’re reading this because we need contributions to make this tool one of the best tool on the Internet!
Submitting changes¶
Before anything, please keep in mind the following. If one or more of those conditions are not filled. Your Pull Request to PyFunceble may not be merged.
The master
branch is used only for releasing a new or stable version of the code. That’s why we require that all contributions/modifications must be done under the dev
or a new branch.
In order to gain some time and also understand what you are working on, your pull requests submission as long as your commit descriptions should be clear and complete as much as possible. We do an exception to commit with minor changed but big changes should have a complete description. Please ensure to use the following method when committing a big change.
Steps before commit¶
Note
The following do not apply if you do not touch the PyFunceble
nor the tests
directory.
$ # We format our code.
$ black PyFunceble && black tests/*.py
$ # We lint our code. Please make sure to fix all reported issues.
$ pylint PyFunceble && pylint tests/*.py
$ # We check the tests coverage. Please ensure that at lease 95% of the code is covered.
$ coverage run setup.py test && coverage report -m
$ # Prepare our files, :code:`version.yaml` and code for pushing.
$ PyFunceble --production
The commit¶
$ # There paragraph is optional if your changes/commits are obvious.
$ git commit -S -m "A summary of the commit" -m "A paragraph
> or a sentence explaining what changed, why and its impact."
All your commits should be signed with PGP. (More information can be found on GitHub documentation)
Please note the usage of -S
into the commit command which means that we sign the commit.
The usage of PyFunceble --production
update version.yaml
and directory_structure_production.json
automatically.
Coding conventions¶
- We make sure that a method, a function, and a class have doctrings.
- One line should not exceed 79 characters for docstring and 100 characters for long declaration/assignment. - Exception granted for regular expressions or long string assignment.
- We use Black, The uncompromising Python code formatter, to format our code.
- Our code should pass
pylint PyFunceble && pylint tests/*.py
with at least a score of 10.00/10.00 - We do not forget to follow the steps before any commits.
Contributors¶
Thanks to those awesome peoples for their awesome and crazy idea(s), contribution(s) and or issue report which made or make PyFunceble a better tool.
_______ _ _ _ _
|__ __| | | | | | | |
| | | |__ __ _ _ __ | | _____ | |_ ___ _ _ ___ _ _ | |
| | | '_ \ / _` | '_ \| |/ / __| | __/ _ \ | | | |/ _ \| | | | | |
| | | | | | (_| | | | | <\__ \ | || (_) | | |_| | (_) | |_| | |_|
|_| |_| |_|\__,_|_| |_|_|\_\___/ \__\___/ \__, |\___/ \__,_| (_)
__/ |
|___/
- dnmTX - @dnmTX
- jawz101 - @jawz101
- Mitchell Krog - @mitchellkrogza
- Odyseus - @Odyseus
- Reza Rizqullah - @ybreza
- ScriptTiger - @ScriptTiger
- speedmann - @speedmann
- WaLLy3K - @WaLLy3K
- xxcriticxx - @xxcriticxx
Special Thanks¶
Thanks to those awesome organization(s), tools and or people(s) for
- Their awesome documentation
- Their awesome repository
- Their awesome tool/software/source code
- Their breaking reports
- Their contributions
- Their current work/purpose
- Their promotion of Py-Funceble
- Their support
- Their testings reports
which helped and/or still help me build, test and or make PyFunceble a better tool.
_______ _ _ _ _
|__ __| | | | | | | |
| | | |__ __ _ _ __ | | _____ | |_ ___ _ _ ___ _ _ | |
| | | '_ \ / _` | '_ \| |/ / __| | __/ _ \ | | | |/ _ \| | | | | |
| | | | | | (_| | | | | <\__ \ | || (_) | | |_| | (_) | |_| | |_|
|_| |_| |_|\__,_|_| |_|_|\_\___/ \__\___/ \__, |\___/ \__,_| (_)
__/ |
|___/
- Adam Warner - @PromoFaux
- Adblock Plus - @adblockplus
- Mitchell Krog - @mitchellkrogza
- Mohammad Fares - @faressoft
- Pi-Hole - @pi-hole
- Public Suffix List - @publicsuffix
- Reza Rizqullah - @ybreza
- ScriptTiger - @ScriptTiger
- SMed79 - @SMed79
- IANA - ICANN
- yEd Graph Editor - yWorks
Supporting the project¶
PyFunceble, Dead-Hosts, and all other analog projects are powered by free time and a lot of coffee!
This project helps you and/or you like it?
They use PyFunceble!¶
Just a list of tool/repository/code/infrastructure which uses or used PyFunceble!
- .pl-host-file
- add.2o7Net
- add.Dead
- add.Risk
- add.Spam
- adfilt
- Badd-Boyz-Hosts
- Badger Sett
- blacklist
- CoinBlockerLists
- cookies_filters
- Facebook Zero Hosts Block
- finnish-easylist-addition
- Frellwits-filter-lists
- Google AdService and ID.Google tracking hosts
- KADhosts
- MobileAdTrackers
- Phishing-URL-Testing-Database-of-Link-Statuses
- Phishing.Database
- polish-adblock-filters
- polish-pihole-filters
- Stop.Google.Analytics.Ghost.Spam.HOWTO
- The-Big-List-of-Hacked-Malware-Web-Sites
- Top-Attacking-IP-Addresses-Against-Wordpress-Sites
- Ultimate.Hosts.Blacklist
- UncheckyAds
License¶
MIT License
Copyright (c) 2017-2018 Nissar Chababy
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Code of Conduct¶
Our Pledge¶
In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.
Our Standards¶
Examples of behavior that contributes to creating a positive environment include:
- Using welcoming and inclusive language
- Being respectful of differing viewpoints and experiences
- Gracefully accepting constructive criticism
- Focusing on what is best for the community
- Showing empathy towards other community members
Examples of unacceptable behavior by participants include:
- The use of sexualized language or imagery and unwelcome sexual attention or advances
- Trolling, insulting/derogatory comments, and personal or political attacks
- Public or private harassment
- Publishing others’ private information, such as a physical or electronic address, without explicit permission
- Other conduct which could reasonably be considered inappropriate in a professional setting
Our Responsibilities¶
Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.
Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.
Scope¶
This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.
Enforcement¶
Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at contact@funilrys.com. The project team will review and investigate all complaints, and will respond in a way that it deems appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.
Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project’s leadership.
Attribution¶
This Code of Conduct is adapted from the Contributor Covenant, version 1.4, available at http://contributor-covenant.org/version/1/4.
Discovered any issue?¶
Want to make it public? Then fill a new issue.
Want to talk with @funilrys about it? Then stay in touch with Fun Ilrys on Keybase (preferred) or send an email to contactTATATAfunilrysTODTOTODcom.
Don’t want to fill an issue on GitHub? You can still inform us about it on our issues team on Keybase.
Need help?¶
Do you need help with the usage or application of PyFunceble? We can help!
Want to talk with @funilrys about it? Then stay in touch with Fun Ilrys on Keybase (preferred) or send an email to contactTATATAfunilrysTODTOTODcom.
Otherwise, let’s talk about it on our support team or our main team on Keybase!