Usage¶
From a terminal¶
Detailed¶
Note
False
stand for deactivated when True
stand for activated.
-ad
| --adblock
¶
Switch the decoding of the adblock format.
Default value:
False
If this argument is activated the system will extract all domains or IP from the given adblock file.
-a
| --all
¶
Output all available information on the screen.
Default value:
False
When activated:
Domain Status Expiration Date Source HTTP Code
---------------------------------------------------------------------------------------------------- ----------- ----------------- ---------- ----------
pyfunceble.readthedocs.io ACTIVE Unknown NSLOOKUP 302
When deactivated:
Domain Status HTTP Code
---------------------------------------------------------------------------------------------------- ----------- ----------
pyfunceble.readthedocs.io ACTIVE 302
-c
| --auto-continue
| --continue
¶
Switch the value of the auto continue mode.
Default value:
True
This argument activates or deactivates the auto-continue subsystem. Indeed, as we can automatically continue if the script has been stopped, this switch allows us to disable or enable the usage of that specific subsystem.
--clean
¶
Clean all files under output.
As it is sometimes needed to clean our output/
directory, this argument does the job automatically.
Warning
This argument delete everything which are .keep
or .gitignore
--clean-all
¶
Clean all files under output and all file generated by PyFunceble.
Warning
This really deletes everything we generated without any warning.
--cmd
“something”¶
Pass a command before each commit (except the final one).
Default value:
''
Note
In this example, something
should be a script or a program which have to be executed when we reached the end of the given file.
Note
This argument is only used if --travis
or travis : true
(under .PyFunceble.yaml
) are activated.
--cmd-before-end "something"
¶
Pass a command before the results (final) commit under the Travis mode.
Default value:
''
Note
In this example, something
should be a script or a program which have to be executed when we reached the end of the given file.
Note
This argument is only used if --travis
or travis : true
(under .PyFunceble.yaml
) are activated.
--commit-autosave-message "something"
¶
Replace the default autosave commit message.
Default value:
PyFunceble - AutoSave
This argument allows us to set a custom commit message which is going to be used as commit message when saving.
Note
This argument is only used if --travis
or travis : true
(under .PyFunceble.yaml
) are used.
Note
This argument is only used if we have to split the work into multiple processes because a list is too long or the timeout is reached.
Warning
Please avoid the usage of [ci skip]
here.
--commit-results-message "something"
¶
Replace the default results (final) commit message.
Default value:
PyFunceble - Results
Note
This argument is only used if --travis
or travis : true
(under .PyFunceble.yaml
) are used.
Note
This argument is only used if we reached the end of the list we are or have to test.
-d "something"
| --domain "something"
¶
Set and test the given domain.
This argument will test and give the results of the tests of the given domain.
Note
For this argument (and only for this argument), we are converting the given string to lowercase.
-db
| --database
¶
Switch the value of the usage of a database to store inactive domains of the currently tested list.
Default value:
True
This argument will disable or enable the usage of a database which saves all INACTIVE and INVALID domain of the given file over time.
Note
The database is retested every x day(s), where x is the number set in -dbr "something"
.
-dbr "something"
¶
Set the numbers of days between each retest of domains present into the database of INACTIVE and INVALID domains.
Default value:
1
Note
This argument is only used if -db
or inactive_database : true
(under .PyFunceble.yaml
) are activated.
--debug
¶
Switch the value of the debug mode.
Default value:
False
This argument activates the debug mode. Under the debug mode, everything caught by the whois subsystem is saved.
Warning
Do not use this argument unless you have been told to.
--directory-structure
¶
Generate the directory and files that are needed and which does not exist in the current directory.
Want to start without anything? This argument generates the output directory automatically for you!
Note
In case of a file or directory not found issue, it’s recommended to remove the dir_structure.json
along with the output/ directory before using this argument.
-ex
| --execution
¶
Switch the default value of the execution time showing.
Default value:
False
Want to know the execution time of your test? Well, this argument will let you know!
-f "something"
| --file "something"
¶
Read the given file and test all domains inside it. If a URL is given we download and test the content of the given URL.
Note
We consider one line as one domain or one commented line. A line can be commented at the end.
Note
You can give a raw link and the system will download and test its content.
--filter "something"
¶
Domain to filter (regex).
Want to test all blogspot
from your list? This argument allows you to do that!
Note
This argument should be a regex expression.
--help
¶
Show the help message and exit.
-h
| --host
¶
Switch the value of the generation of hosts file.
Default value:
True
This argument will let the system know if it has to generate the hosts file version of each status.
--hierarchical
¶
Switch the value of the hierarchical sorting of the tested file.
Default value:
True
This argument will let the system know if we have to sort the list and our output in hierarchical order.
--http
¶
Switch the value of the usage of HTTP code.
Default value:
True
You don’t want to take the result of the HTTP code execution in consideration? This argument allows you to disable that!
Note
If activated the subsystem will bypass the HTTP status code extraction logic-representation.rst
--iana
¶
Update/Generate iana-domains-db.json.
This argument generates or updates iana-domains-db.json.
--idna
¶
Switch the value of the IDNA conversion.
Default value:
False
This argument allows the conversion of the domains using domain2idna
Warning
This feature is not supported yet for the URL testing.
-ip "something"
¶
Change the IP to print with the hosts files.
Default value:
0.0.0.0
--json
¶
Switch the value of the generation of the JSON formatted list of domains.
Default value:
False
--less
¶
When activated:
Domain Status HTTP Code
---------------------------------------------------------------------------------------------------- ----------- ----------
pyfunceble.readthedocs.io ACTIVE 302
When deactivated:
Domain Status Expiration Date Source HTTP Code
---------------------------------------------------------------------------------------------------- ----------- ----------------- ---------- ----------
pyfunceble.readthedocs.io ACTIVE Unknown NSLOOKUP 302
--local
¶
Switch the value of the local network testing.
Default value:
False
Want to run a test over a local or private network? This argument will disable the limitation which does not apply to private networks.
--link "something"
¶
Download and test the given file.
Want to test a raw link? This argument will download and test the given raw link.
-m
| --mining
¶
Switch the value of the mining subsystem usage.
Default value:
False
Want to find domain or URL linked to a domain in your list? This argument will exactly do that.
-n
| --no-files
¶
Switch the value the production of output files.
Default value:
False
Want to disable the production of the outputted files? This argument is for you!
-nl
| --no-logs
¶
Switch the value of the production of logs files in the case we encounter some errors.
Default value:
False
Don’t want any logs to go out of PyFunceble? This argument disables every logs subsystem.
-nu
| --no-unified
¶
Switch the value of the production unified logs under the output directory.
Default value:
True
This argument disables the generation of result.txt.
-nw
| --no-whois
¶
Switch the value the usage of whois to test domain’s status.
Default value:
False
Don’t want to use or take in consideration the results from whois
? This argument allows you to disable it!
-p
| --percentage
¶
Switch the value of the percentage output mode.
Default value:
True
This argument will disable or enable the generation of the percentage of each status.
--plain
¶
Switch the value of the generation of the plain list of domains.
Default value:
False:
Want to get a list with all domain for each status? The activation of this argument does the work while testing!
--production
¶
Prepare the repository for production.
Warning
Do not use this argument unless you have been told to, you prepare a Pull Request or you want to distribute your modified version of PyFunceble.
-psl
| --public-suffix
¶
Update/Generate public-suffix.json.
This argument will generate or update public-suffix.json.
-q
| --quiet
¶
Run the script in quiet mode.
Default value:
False
You prefer to run a program silently? This argument is for you!
-s
| --simple
¶
Switch the value of the simple output mode.
Default value:
False
Want as less as possible data on screen? This argument returns as less as possible on screen!
--split
¶
Switch the value of the split of the generated output
Default value:
True
Want to get the logs (copy of what you see on screen) on different files? This argument is suited to you!
--syntax
¶
Switch the value of the syntax test mode.
Default value:
False
-t "something"
| --timeout "something"
¶
Switch the value of the timeout.
Default value:
3
This argument will set the default timeout to apply everywhere it is possible to set a timeout.
--travis
¶
Switch the value of the Travis mode.
Default value:
False
Want to use PyFunceble under Travis CI? This argument is suited for your need!
-url "something"
| --url "something"
¶
Analyze the given URL.
Want to test the availability or an URL? Enjoy this argument!
Note
When we test the availability of an URL, we check the HTTP status code of the given URL.
-uf "something"
| --url-file "something"
¶
Read and test the list of URL of the given file. If a URL is given we download and test the content of the given URL.
Note
We consider one line as one URL to test.
Note
You can give a raw link and the system will download and test its content.
-ua "something"
| --user-agent "something"
¶
Set the user-agent to use and set every time we interact with everything which is not our logs sharing system.
-v
| --version
¶
Show the version of PyFunceble and exit.
-vsc
| --verify-ssl-certificate
¶
Switch the value of the verification of the SSL/TLS certificate when testing for URL.
Default value:
False
Warning
If you activate the verification of the SSL/TLS certificate, you may get false positive results.
Indeed if the certificate is not registered to the CA or is simply invalid and the domain is still alive, you will always get
INACTIVE
as output.
-wdb
| --whois-database
¶
Switch the value of the usage of a database to store whois data in order to avoid whois servers rate limit.
Default value:
True
Global overview¶
usage: PyFunceble [-ad] [-a] [-c] [--autosave-minutes AUTOSAVE_MINUTES]
[--clean] [--clean-all] [--cmd CMD]
[--cmd-before-end CMD_BEFORE_END]
[--commit-autosave-message COMMIT_AUTOSAVE_MESSAGE]
[--commit-results-message COMMIT_RESULTS_MESSAGE]
[-d DOMAIN] [-db] [-dbr DAYS_BETWEEN_DB_RETEST] [--debug]
[--directory-structure] [-ex] [-f FILE] [--filter FILTER]
[--help] [--hierarchical] [-h] [--http] [--iana] [--idna]
[-ip IP] [--json] [--less] [--local] [--link LINK] [-m] [-n]
[-nl] [-nu] [-nw] [-p] [--plain] [--production] [-psl] [-q]
[--share-logs] [-s] [--split] [--syntax] [-t TIMEOUT]
[--travis] [--travis-branch TRAVIS_BRANCH] [-u URL]
[-uf URL_FILE] [-ua USER_AGENT] [-v] [-vsc] [-wdb]
optional arguments:
-ad, --adblock Switch the decoding of the adblock format.
Configured value: False
-a, --all Output all available informations on screen.
Configured value: True
-c, --auto-continue, --continue
Switch the value of the auto continue mode.
Configured value: True
--autosave-minutes AUTOSAVE_MINUTES
Update the minimum of minutes before we start
committing to upstream under Travis CI.
Configured value: 15
--clean Clean all files under output.
--clean-all Clean all files under output and all file generated by
PyFunceble.
--cmd CMD Pass a command to run before each commit (except the
final one) under the travis mode. Configured
value: ''
--cmd-before-end CMD_BEFORE_END
Pass a command to run before the results (final)
commit under the Travis mode. Configured
value: ''
--commit-autosave-message COMMIT_AUTOSAVE_MESSAGE
Replace the default autosave commit message.
Configured value: 'PyFunceble -
AutoSave'
--commit-results-message COMMIT_RESULTS_MESSAGE
Replace the default results (final) commit message.
Configured value: 'PyFunceble -
Results'
-d DOMAIN, --domain DOMAIN
Set and test the given domain.
-db, --database Switch the value of the usage of a database to store
inactive domains of the currently tested list.
Configured value: True
-dbr DAYS_BETWEEN_DB_RETEST, --days-between-db-retest DAYS_BETWEEN_DB_RETEST
Set the numbers of days between each retest of
domains present into inactive-db.json.
Configured value: 1
--debug Switch the value of the debug mode.
Configured value: False
--directory-structure
Generate the directory and files that are needed and
which does not exist in the current directory.
-ex, --execution Switch the default value of the execution time
showing. Configured value: False
-f FILE, --file FILE Read the given file and test all domains inside it. If
a URL is given we download and test the content of the
given URL.
--filter FILTER Domain to filter (regex).
--help Show this help message and exit.
--hierarchical Switch the value of the hierarchical sorting of tested
file. Configured value: True
-h, --host Switch the value of the generation of hosts file.
Configured value: True
--http Switch the value of the usage of HTTP code.
Configured value: True
--iana Update/Generate `iana-domains-db.json`.
--idna Switch the value of the IDNA conversion.
Configured value: False
-ip IP Change the IP to print in the hosts files with the given one.
Configured value: '0.0.0.0'
--json Switch the value of the generation of the json list of
domain. Configured value: False
--less Output less informations on screen.
Configured value: False
--local Switch the value of the local network testing.
Configured value: True
--link LINK Download and test the given file.
-m, --mining Switch the value of the mining subsystem usage.
Configured value: False
-n, --no-files Switch the value of the production of output files.
Configured value: False
-nl, --no-logs Switch the value of the production of logs files in
the case we encounter some errors. Configured
value: False
-nu, --no-unified Switch the value of the production unified logs under
the output directory. Configured value:
True
-nw, --no-whois Switch the value the usage of whois to test domain's
status. Configured value: False
-p, --percentage Switch the value of the percentage output mode.
Configured value: True
--plain Switch the value of the generation of the plain list
of domain. Configured value: False
--production Prepare the repository for production.
-psl, --public-suffix
Update/Generate `public-suffix.json`.
-q, --quiet Run the script in quiet mode. Configured
value: False
--share-logs Switch the value of the sharing of logs.
Configured value: True
-s, --simple Switch the value of the simple output mode.
Configured value: False
--split Switch the value of the split of the generated output
files. Configured value: True
--syntax Switch the value of the syntax test mode.
Configured value: False
-t TIMEOUT, --timeout TIMEOUT
Switch the value of the timeout. Configured
value: 3
--travis Switch the value of the Travis mode.
Configured value: False
--travis-branch TRAVIS_BRANCH
Switch the branch name where we are going to push.
Configured value: 'master'
-u URL, --url URL Analyze the given URL.
-uf URL_FILE, --url-file URL_FILE
Read and test the list of URL of the given file. If a
URL is given we download and test the content of the
given URL.
-ua USER_AGENT, --user-agent USER_AGENT
Set the user-agent to use and set every time we
interact with everything which is not our logs sharing
system.
-v, --version Show the version of PyFunceble and exit.
-vsc, --verify-ssl-certificate
Switch the value of the verification of the SSL/TLS
certificate when testing for URL. Configured
value: False
-wdb, --whois-database
Switch the value of the usage of a database to store
whois data in order to avoid whois servers rate limit.
Configured value: True
Crafted with ♥ by Nissar Chababy (Funilrys) with the
help of https://pyfunceble.rtfd.io/en/master/contributors.html &&
https://pyfunceble.rtfd.io/en/master/special-thanks.html
From a Python script or module¶
Before continuing reading this part, You should know that I consider that you can speak in Python. If it’s not the case, well, it’s the time to learn Python!
As PyFunceble is written in Python, it can be easily imported and used inside a script. This part will represent a basic example of usage.
Basic example¶
"""
This is a basic example which prints one of the official output of PyFunceble.
Note:
* Official output: ACTIVE, INACTIVE, INVALID
"""
from PyFunceble import test as PyFunceble
from PyFunceble import url_test as PyFuncebleURL
print(PyFunceble(domain='google.com'))
print(PyFuncebleURL(url='https://google.com'))
"""
This is a basic example which checks syntax.
"""
from PyFunceble import syntax_check as PyFuncebleDomainSyntax
from PyFunceble import url_syntax_check as PyFuncebleURLSyntax
from PyFunceble import ipv4_syntax_check as PyFuncebleIPv4Syntax
print("google.com", PyFuncebleDomainSyntax(domain="google.com"))
print("https://google.com", PyFuncebleURLSyntax(url="https://google.com"))
print("216.58.207.46", PyFuncebleIPv4Syntax(ip="216.58.207.46"))
print("forest-jump", PyFuncebleDomainSyntax(domain="forest-jump"))
print("https://forest-jump", PyFuncebleURLSyntax(url="https://forest-jump"))
print("257.58.207.46", PyFuncebleIPv4Syntax(ip="257.58.207.46"))
Loop example¶
This part is unnecessary but we wanted to document it!!
"""
This is a loop example which tests a list of domain and processes some action
according to one of the official output of PyFunceble.
Note:
* Official output: ACTIVE, INACTIVE, INVALID
* You should always use PyFunceble().test() as it's the method which is especially
suited for `__name__ != '__main__'` usage.
"""
from PyFunceble import test as PyFunceble
from PyFunceble import url_test as PyFuncebleURL
DOMAINS = ["twitter.com", "google.com", "github.com", "github.comcomcom", "funilrys.co"]
def domain_status(domain_or_ip):
"""
Check the status of the given domain name or IP.
Argument:
- domain_or_ip: str
The domain or IPv4 to test.
Returns: str
The status of the domain.
"""
return PyFunceble(domain_or_ip)
def url_status(url):
"""
Check the status of the given url.
Argument:
- url: str
The URL to test.
Returns: str
The status of the URL.
"""
return PyFuncebleURL(url)
for domain in DOMAINS:
print(
"%s is %s and %s is %s"
% (
domain,
domain_status(domain),
"http://" + domain,
url_status("http://" + domain),
)
)
Advanced example¶
PyFunceble now allow you to get the following information as a dictionary. The objective behind this feature is to let you know more about the element you are testing.
{
"tested": None, # The tested element.
"expiration_date": None, # The expiration_date of the element if found.
"domain_syntax_validation": None, # The domain syntax validation status.
"http_status_code": None, # The status code of the tested element.
"ip4_syntax_validation": None, # The IPv4 syntax validation status.
"nslookup": [], # A list of IP of the tested element.
"status": None, # The status matched by PyFunceble.
"url_syntax_validation": None, # The url syntax validation status.
"whois_server": None, # The whois server if found.
"whois_record": None, # The whois record if whois_server is found.
}
To get that information simply work with our interface like follow :)
"""
This is an advanced example which prints some information about the tested element.
Note:
* Official output: ACTIVE, INACTIVE, INVALID
"""
from PyFunceble import test as PyFunceble
from PyFunceble import url_test as PyFuncebleURL
domain_testing = PyFunceble(domain='google.com', complete=True)
url_testing = PyFuncebleURL(url='https://google.com', complete=True)
print(domain_testing['nslookup'])
print(domain_testing['domain_syntax_validation'])
print(domain_testing['domain'], domain_testing['status'])
print(url_testing['nslookup'])
print(url_testing['domain_syntax_validation'])
print(url_testing['domain'], domain_testing['status'])
From Travis CI¶
As we offer an argument named --travis
to activate the usage of PyFunceble in a Travis CI instance, we document here what you need to know!
Configuration¶
Note
This part only present a commented .travis.yml
so that you can understand where to start.
If you need more practical examples, feel free to report to one of Dead-Hosts repositories which use PyFunceble with Travis CI.
env:
global:
# The following is your encrypted GitHub API key.
# Indeed as we are going to push to the repository, this is needed.
- secure: QQdKFquFFojFT9XJ1XZp4EMoDTVoXFgqZq8XU+sCVf+pJQR6d/oKBp8rnSTCnZizWOQXUjGXUUxUpSG/dYGyBLjo3rH3rsn9ciZHVfubxbwK860w4sqibl4DvhCv2rdsFtvzXnhm4P9OL3i+krKdewh9fxpNyUU58qOgfnS7mK9FcFhb8z5ak2sxU2XRZedwm6Ro0oyVKs8kFkL4YaADfNyAHlGTfr9rVmE52WXQXQENktb9gFgR2A8ZnmLy0BCMZGkPDShJnjRDWD4DErtasLmLQvWpzOBwdbVJTY6U9KDRXVNdC9lp5E5Ba/dc0y36q6vjfgJR+QchetOtHgNbKYbLB8c26Di90OZCFJsxMNcl1Wct4qFPXkFGvjXrISW6pbdPL5Plto0Ig3iLiulhYOPVArysMIk9ymtSXP+WE7VWX01LQ1fEkIoSfeVZ2caTnCmTsoHVGRRe978CojKaT7yU45kb15hcyDrzptQ8EP2hfxeh5F7KtueQ6Rsb9LFDZMkMDKflZn6a+bRhESlmWWmYB9stzGzTurQA1E1bcSACJ8A8hG5nHBzZYJ2S+OY0PE7UdyOJ0JK0qe/67d+F9ocQdIoFpDDTdgIjHerQnD2wRg1aKPzLDb4jJTpqgr5ssPrqUAKl3st7gyaAZzCEADPDnIBDjOJS+mFWbx9DKgc=
# This is the Git name we have to set. (git config user.name)
- GIT_NAME: Travis CI
# This is the Git Email we have to set. (git config user.email)
- GIT_EMAIL: dead-hosts@funilrys.com
# This is the full slug of the repository we are working with.
- TRAVIS_REPO_SLUG: dead-hosts/repository-structure
# This is the branch we have to checkout and push to.
- GIT_BRANCH: master
# This is the language we use.
language: python
# This is the python version we are going to use for the tests.
# Note: you can add any 3.x version to the list.
python:
- "3.6"
# The following will tell Travis CI to ends as fast as possible.
matrix:
fast_finish: true
# Here we are setting what Travis CI have to cache.
cache:
# We are caching pip3 as we use it to install PyFunceble
- pip3
install:
# We install the development version of PyFunceble. If you prefer the stable version replace `pyfunceble-dev` with `pyfunceble`.
- pip3 install pyfunceble-dev
# Our tests start here.
script:
# Let's say we want our results and our PyFunceble infrastructure to be saved in a directory called `PyFunceble-tests`
# We move inside it.
- cd PyFunceble-tests
# We test the file `my_awesome_list` which is located inside the current directory.
# Note: we precise the `--travis` argument here,
# but you work without it if you set `travis: true` inside your `.PyFunceble.yaml`
- PyFunceble --travis -f my_awesome_list --plain
# The following initiate email notification logic.
notifications:
# As we want to get a mail on failure and on status change, we set the following.
on_success: change
on_failure: always
Getting a GitHub token¶
For the secure
index of the .travis.yml
file, you have to generate a new GitHub token.
After you got your token, please write it or save it in a safe place as you’re going to need it every time you’re going to interact with Travis CI.
Note
The scope to set is public_repo
but you can also set others depending on your needs.
Encrypting the token for Travis CI usage¶
To encrypt the token simply replace and execute the following according to your personal case.
$ travis encrypt 'GH_TOKEN=theGeneratedToken' -r 'The content of TRAVIS_REPO_SLUG' --add
Warning
Please do not execute the following explicitly without replacing theGeneratedToken
with your previously generated GitHub token and The content of TRAVIS_REPO_SLUG
with your repository slug.
Note
The usage of --add
ensure that the travis
program automatically add the secure
index to the .travis.yml
file.