Usage¶
From a terminal¶
--help
¶
Show the help message and exit.
-v
| --version
¶
Show the version of PyFunceble and exit.
Source¶
-d "something"
| --domain "something"
¶
Test one or more domains, separated by spaces.
Note
This argument takes 1 or more values.
As example:
$ PyFunceble -d example.org example.net
Note
When this option is used, no output files are generated.
-url "something"
| --url "something"
¶
Test one or more full URL, separated by spaces.
Note
When we test the availability of a URL, we (only) check the HTTP status code of the given URL.
Note
This argument takes 1 or more values.
As example:
$ PyFunceble -u https://example.org https://example.com
-f "something"
| --file "something"
¶
Read a local or remote (RAW link) file and test all domains inside it. If remote (RAW link) file is given, PyFunceble will download it, and test the content of the given RAW link as if it was a locally stored file.
Note
The system understands the following format:
- plain list of subjects.
- hosts file format.
-uf "something"
| --url-file "something"
¶
Read a local or remote (RAW link) file and test all (full) URLs inside it. If remote (RAW link) file is given, PyFunceble will download it, and test the content of the given RAW link as if it was a locally stored file.
Note
We consider one line as one URL to test.
Note
This argument test if a URL which is inside the given file is available. It ONLY tests full URLs.
As example:
$ PyFunceble -uf `https://raw.githubusercontent.com/funilrys/PyFunceble/master/.travis/lists/url`
will download the given URL and test for its content assuming that each line represents a URL to test.
Warning
A test with this argument consists of the comparison of the status code. No WHOIS record will be requested nor DNS Lookup will be done.
Source filtering, decoding, conversion and expansion¶
-ad
| --adblock
¶
Switch the decoding of the adblock format.
Default value:
False
If this argument is activated the system will extract all domains or IP from the given adblock file.
--complements
¶
Switch the value of the generation and test of the complements. A complement is for example
example.org
ifwww.example.org
is given and vice-versa.Default value:
False
--filter "something"
¶
Domain to filter (regex).
Want to test all blogspot
from your list? This argument allows you to
do that!
Note
This argument should be a regex expression.
--idna
¶
Switch the value of the IDNA conversion.
Default value:
False
This argument allows the conversion of the domains using domain2idna
--mining
¶
Switch the value of the mining subsystem usage.
Default value:
False
Want to find domain or URL linked to a domain in your list? This argument will exactly do that.
Test control¶
-c
| --auto-continue
| --continue
¶
Switch the value of the auto continue mode.
Default value:
True
This argument activates or deactivates the auto-continue subsystem. Indeed, as we can automatically continue if the script has been stopped, this switch allows us to disable or enable the usage of that specific subsystem.
--cooldown-time
¶
Switch the value of the cool-down time to apply between each test.
Default value:
None
This argument applies a number of seconds to sleep before/between each test.
--http
¶
Switch the value of the usage of HTTP code.
Default value:
True
You don’t want to take the result of the HTTP code execution into consideration? This argument allows you to disable that!
Note
If activated the subsystem will bypass the HTTP status code extraction logic-representation.rst
--local
¶
Switch the value of the local network testing.
Default value:
False
Want to run a test over a local or private network? This argument will disable the limitation which does not apply to private networks.
-ns
| --no-special
¶
Switch the value of the usage of the SPECIAL rules.
Default value:
False
Don’t want to use/apply the SPECIAL rules - which are explained in the source column section? This argument disables them all.
-nw
| --no-whois
¶
Switch the value of the usage of WHOIS to test the domain’s status.
Default value:
False
Don’t want to use or take into consideration the results from whois
?
This argument allows you to disable it!
--reputation
¶
Switch the value of the reputation test mode.
Default value:
False
Warning
This will disable all other forms of test,
will check against AlienVault’s reputation data
and output its result into output/*/{MALICIOUS,SANE}/*
.
--shadow-file
| --shadow
¶
Switch the value of the usage and generation of a shadow file before a file test starts.
A shadow file is a file which only contain the actual list of subject to test. For its generation we check each subjects as we normally do on-the-fly.
Default value:
False
Note
The shadow file, will just contain the actual list of subjects to test.
--syntax
¶
Switch the value of the syntax test mode.
Default value:
False
Warning
This will disable all other forms of test,
will validate the syntax of a given test subject,
and output its results in plain format into
output/domains/{VALID,INVALID}/list
-t "something"
| --timeout "something"
¶
Switch the value of the timeout in seconds.
Default value:
5
This argument will set the default timeout to apply everywhere it is possible to set a timeout.
--use-reputation-data
¶
Switch the value of the reputation data usage.
Default value:
False
Warning
This only affects when used along with the availability test.
-ua "something"
| --user-agent "something"
¶
Set the user-agent to use and set every time we interact with everything which is not the logs sharing system.
-vsc
| --verify-ssl-certificate
¶
Switch the value of the verification of the SSL/TLS certificate when testing for URL.
Default value:
False
Warning
If you activate the verification of the SSL/TLS certificate, you may get false-positive results.
Indeed if the certificate is not registered to the CA or is simply invalid and the domain is still alive, you will always get
INACTIVE
as output.
wildcard
¶
Switch the value of the wildcards test.
Default value:
False
Warning
This argument is not taken into consideration if the
--syntax
argument is not given.
DNS (resolver) control¶
--dns
¶
Set one or more specific DNS servers to use during the test. Separated by spaces.
Default value:
Follow OS DNS
==>None
Warning
We expect a DNS server(s). If no DNS server(s) is given. You’ll almost for
certain get all results as INACTIVE
This could happen in case you use --dns -f
Note
You can specify a port number to use to the DNS server if needed.
As example:
- 127.0.1.53:5353
--dns-lookup-over-tcp
¶
Make all DNS queries through TCP instead of UDP.
Default value:
False
Databases¶
-db
| --database
¶
Switch the value of the usage of a database to store inactive domains of the currently tested list.
Default value:
True
This argument will disable or enable the usage of a database which saves all
INACTIVE
and INVALID
domain of the given file over time.
Note
The database is retested every x day(s), where x is the number set in
-dbr "something"
.
--database-type
¶
Tell us the type of database to use. You can choose between the following:
json
,mariadb
,mysql
.Default value:
json
This argument let us use different types of database.
Note
This feature is applied to the following subsystems:
- Autocontinue physically located (JSON) at
output/continue.json
. - InactiveDB physically located (JSON) at
[config_dir]/inactive_db.json
. - Mining physically located (JSON) at
[config_dir]/mining.json
. - WhoisDB physically located (JSON) at
[config_dir]/whois.json
.
-dbr "something"
| --days-between-db-retest "something"
¶
Set the numbers of days between each retest of domains present into the database of INACTIVE and INVALID domains.
Default value:
1
Note
This argument is only used if -db
or
inactive_database : true
(under .PyFunceble.yaml
) are
activated.
-dbc "something"
| --days-between-db-clean
¶
Set the numbers of days since the introduction of a subject into inactive-db.json for it to qualifies for deletion.
Default value:
28
Note
This argument is only used if -db
or
inactive_database : true
(under .PyFunceble.yaml
) are
activated.
-wdb
| --whois-database
¶
Switch the value of the usage of a database to store whois data to avoid whois servers rate limit.
Default value:
True
Output control¶
-a
| --all
¶
Output all available information on the screen.
Default value:
False
When activated:
Domain Status Expiration Date Source HTTP Code
----------------------------- ----------- ----------------- ---------- ----------
pyfunceble.readthedocs.io ACTIVE Unknown NSLOOKUP 302
When deactivated:
Domain Status HTTP Code
----------------------------- ----------- ----------
pyfunceble.readthedocs.io ACTIVE 302
--dots
¶
Output dots (
.
) tostdout
instead of giving the impression that we hang on.Default value:
False
-ex
| --execution
¶
Switch the default value of the execution time showing.
Default value:
False
Want to know the execution time of your test? Well, this argument will let you know!
--hierarchical
¶
Switch the value of the hierarchical sorting of the tested file.
Default value:
True
This argument will output the result listed in a hierarchical order.
-h
| --host
¶
Switch the value of the generation of hosts file.
Default value:
True
This argument will let the system know if it has to generate the hosts file version of each status.
-ip "something"
¶
Change the IP to print in the hosts files with the given one.
Default value:
0.0.0.0
--json
¶
Switch the value of the generation of the JSON formatted list of domains.
Default value:
False
--less
¶
When activated:
Domain Status HTTP Code
------------------------------------------------------ ----------- ----------
pyfunceble.readthedocs.io ACTIVE 302
When deactivated:
Domain Status Expiration Date Source HTTP Code
---------------------------- ----------- ----------------- ---------- ----------
pyfunceble.readthedocs.io ACTIVE Unknown NSLOOKUP 302
-nf
| --no-files
¶
Switch the value of the production of output files.
Default value:
False
Want to disable the production of the outputted files? This argument is for you!
-nl
| --no-logs
¶
Switch the value of the production of logs files in the case we encounter some errors.
Default value:
False
Don’t want any logs to go out of PyFunceble? This argument disables every log subsystem.
-nu
| --no-unified
¶
Switch the value of the production unified logs under the output directory.
Default value:
True
This argument disables the generation of result.txt.
--percentage
¶
Switch the value of the percentage output mode.
Default value:
True
This argument will disable or enable the generation of the percentage of each status.
--plain
¶
Switch the value of the generation of the plain list of domains.
Default value:
False:
Want to get a list with all domains for each status? The activation of this argument does the work while testing!
-q
| --quiet
¶
Run the script in quiet mode.
Default value:
False
You prefer to run a program silently? This argument is for you!
-s
| --simple
¶
Switch the value of the simple output mode.
Default value:
False
Want as less as possible data on screen? This argument returns as less as possible on screen!
--split
¶
Switch the value of the split of the generated output.
Default value:
True
Want to get the logs (copy of what you see on screen) on different files? This argument is suited to you!
--store-whois
¶
Switch the value of the WHOIS record storage in the WHOIS DB.
Default value:
False
The difference between False
or True
is whether
we are saving a full dump of the WHOIS reply into the database.
If you for some reason believes you need to fill up your database with a complete dump of the whois reply, this is the right value to switch on.
Warning
Before switching this value, you should read these comments carefully…
You can test the amount of data by running whois mypdns.org
from your Linux terminal, to see an example of what will be stored
in the database.
You’re hearby warned…
Multiprocessing¶
-m
| --multiprocess
¶
Switch the value of the usage of multiple processes.
Default value:
False
Want to speed up the test time? This argument will allow the usage of multiple processes for testing.
--multiprocess-merging-mode
¶
Sets the multiprocess merging mode. You can choose between the following live|ends.
Default value:
end
Note
With the end
value, the merging of cross-process data is made at
the very end of the current instance.
Note
With the live
value, the merging of cross-process data is made
after the processing of the maximal number of processes.
Which means that if you allow 5 processes, we will run 5 tests, merge, run 5 tests, merge and so on until the end.
-p
| --processes
¶
Set the number of simultaneous processes to use while using multiple processes.
Default value:
25
Warning
DO not try to exceed your number of CPU if you want to keep your machine somehow alive and healthy!!
Note
If omitted, the number of available CPU cores will be used instead.
CI / CD¶
--autosave-minutes
¶
Update the minimum of minutes before we start committing to upstream under the CI mode.
Default value:
15
--ci
¶
Switch the value of the CI mode.
Default value:
False
Note
If you combine this argument with the --quiet
argument, the test
will output a dotted line, where each dot (.
) represent one test
result or input which was skipped because it was previously tested.
Want to use PyFunceble under a supporter CI infrastructure/network? This argument is suited for your needs!
--ci-branch
¶
Switch the branch name where we are going to push the temporary results.
Default value:
master
Note
Currently the branch need to exist, but there are being worked on a path
to have PyFunceble to create the sub-branch and finally merge it into the
--ci-distribution-branch
--ci-distribution-branch
¶
Switch the branch name where we are going to push the final results.
Default value:
master
Note
The difference between this and --ci-branch
is the fact
that this branch will get the (final) result only when the test is finished
under the given --ci-branch
.
As an example, this allows us to have 2 branches:
proceessing
(CI branch), for the tests with PyFunceble.master
(CI distribution branch), for the distribution of the results of PyFunceble.
--cmd
“something”¶
Pass a command before each commit (except the final one).
Default value:
''
Note
In this example, something
should be a script or a program which
have to be executed when we reached the end of the given file.
Note
This argument is only used if --ci
or ci: true
(under
.PyFunceble.yaml
) are activated.
--cmd-before-end "something"
¶
Pass a command before the results (final) commit under the CI mode.
Default value:
''
Note
In this example, something
should be a script or a program which
have to be executed when we reached the end of the given file.
Note
This argument is only used if --ci
or ci: true
(under
.PyFunceble.yaml
) are activated.
--commit-autosave-message "something"
¶
Replace the default autosave commit message.
Default value:
PyFunceble - AutoSave
This argument allows us to set a custom commit message which is going to be used as a commit message when saving.
Note
This argument is only used if --ci
or ci: true
(under
.PyFunceble.yaml
) are used.
Note
This argument is only used if we have to split the work into multiple processes because a list is too long or the timeout is reached.
Warning
Please avoid the usage of [ci skip]
here.
--commit-results-message "something"
¶
Replace the default results (final) commit message.
Default value:
PyFunceble - Results
Note
This argument is only used if --ci
or ci: true
(under
.PyFunceble.yaml
) are used.
Note
This argument is only used if we reached the end of the list we are or have to test.
Unique actions¶
--clean
¶
Clean all files under the output directory.
As it is sometimes needed to clean our output/
directory, this
argument does the job automatically.
Warning
This argument delete everything which are .keep
or
.gitignore
--clean-all
¶
Clean all files under the output directory along with all file generated by PyFunceble.
Warning
This deletes almost everything we generated without any warning.
Note
We don’t delete the whois database file/table because they are (almost) static data which are shared across launches in your environment.
Warning
If you plan to clean manually do not delete the whois database file or table as it will make your test finish under a much longer time as usual for you.
Warning
If you don’t combine this argument alongside with the --database-type
argument or its configurable equivalent, this argument will only clean the
JSON formatted databases.
--directory-structure
¶
Generate the directory and files that are needed and which does not exist in the current directory.
Want to start without anything? This argument generates the output directory automatically for you!
Note
In case of a file or directory not found issue, it’s recommended to remove
the dir_structure.json
along with the output/ directory before
using this argument.
Global overview¶
usage: PyFunceble [-d DOMAIN [DOMAIN ...]] [-u URL [URL ...]] [-f FILE]
[-uf URL_FILE] [-ad] [--complements] [--filter FILTER]
[--idna] [--mining] [-c] [--cooldown-time COOLDOWN_TIME]
[--http] [--local] [-ns] [-nw] [--reputation]
[--shadow-file] [--syntax] [-t TIMEOUT]
[--use-reputation-data] [-ua USER_AGENT] [-vsc] [--wildcard]
[--dns DNS [DNS ...]] [--dns-lookup-over-tcp] [-db]
[--database-type DATABASE_TYPE]
[-dbr DAYS_BETWEEN_DB_RETEST] [-dbc DAYS_BETWEEN_DB_CLEAN]
[-wdb] [-a] [-ex] [--hierarchical] [-h] [-ip IP] [--json]
[--less] [-nf] [-nl] [-nu] [--percentage] [--plain] [--dots]
[-q] [--share-logs] [-s] [--split] [--store-whois] [-m]
[--multiprocess-merging-mode MULTIPROCESS_MERGING_MODE]
[-p PROCESSES] [--autosave-minutes AUTOSAVE_MINUTES] [--ci]
[--ci-branch CI_BRANCH]
[--ci-distribution-branch CI_DISTRIBUTION_BRANCH]
[--cmd CMD] [--cmd-before-end CMD_BEFORE_END]
[--commit-autosave-message COMMIT_AUTOSAVE_MESSAGE]
[--commit-results-message COMMIT_RESULTS_MESSAGE] [--clean]
[--clean-all] [--directory-structure] [--help] [-v]
PyFunceble - The tool to check the availability or syntax of domain, IP or URL.
optional arguments:
--help Show this help message and exit.
-v, --version Show the version of PyFunceble and exit.
Source:
-d DOMAIN [DOMAIN ...], --domain DOMAIN [DOMAIN ...]
Test one or more domains, separated by spaces.
When this option is used, no output files are generated.
-u URL [URL ...], --url URL [URL ...]
Test one or more full URL, separated by spaces.
-f FILE, --file FILE Read a local or remote (RAW link) file and test all domains inside it.
If remote (RAW link) file is given, PyFunceble will download it,
and test the content of the given RAW link as if it was a locally stored file.
-uf URL_FILE, --url-file URL_FILE
Read a local or remote (RAW link) file and test all (full) URLs inside it.
If remote (RAW link) file is given, PyFunceble will download it,
and test the content of the given RAW link as if it was a locally stored file.
This argument test if an URL is available. It ONLY test full URLs.
Source filtering, decoding, conversion and expansion:
-ad, --adblock Switch the decoding of the adblock format.
Configured value: False
--complements Switch the value of the generation and test of the complements.
A complement is for example `example.org` if `www.example.org` is given and vice-versa.
Configured value: False
--filter FILTER Domain to filter (regex).
--idna Switch the value of the IDNA conversion.
Configured value: False
--mining Switch the value of the mining subsystem usage.
Configured value: False
Test control:
-c, --auto-continue, --continue
Switch the value of the auto continue mode.
Configured value: True
--cooldown-time COOLDOWN_TIME
Switch the value of the cooldown time to apply between each test.
Configured value: None
--http Switch the value of the usage of HTTP code.
Configured value: True
--local Switch the value of the local network testing.
Configured value: True
-ns, --no-special Switch the value of the usage of the SPECIAL rules.
Configured value: False
-nw, --no-whois Switch the value of the usage of WHOIS to test the domain's status.
Configured value: False
--reputation Switch the value of the reputation test mode.
Configured value: False
--shadow-file, --shadow
Switch the value of the usage and generation of a shadow file before a file test starts.
A shadow file is a file which only contain the actual list of subject to test. For its generation we check each subjects as we normally do on-the-fly.
Configured value: False
--syntax Switch the value of the syntax test mode.
Configured value: False
-t TIMEOUT, --timeout TIMEOUT
Switch the value of the timeout in seconds.
Configured value: 5
--use-reputation-data
Switch the value of the reputation data usage.
Configured value: False
-ua USER_AGENT, --user-agent USER_AGENT
Set the user-agent to use and set every time we interact with everything which
is not the logs sharing system.
-vsc, --verify-ssl-certificate
Switch the value of the verification of the SSL/TLS certificate when testing for URL.
Configured value: False
--wildcard Switch the value of the wildcards test.
When used, wildcards will be proprely tested.
Warning: This is not taken in consideration if the '--syntax' argument is not given.
Configured value: False
DNS (resolver) control:
--dns DNS [DNS ...] Set one or more DNS server(s) to use during testing. Separated by spaces.
To specify a port number for the DNS server you append it as :port [ip:port].
If no port is specified, the default DNS port (53) is used.
Configured value: OS (declared) DNS server
--dns-lookup-over-tcp
Make all DNS queries with TCP.
Configured value: False
Databases:
-db, --database Switch the value of the usage of a database to store inactive domains of the currently tested list.
Configured value: True
--database-type DATABASE_TYPE
Tell us the type of database to use.
You can choose between the following: `json | mariadb | mysql`
Configured value: 'json'
-dbr DAYS_BETWEEN_DB_RETEST, --days-between-db-retest DAYS_BETWEEN_DB_RETEST
Set the numbers of days between each retest of domains present into inactive-db.json.
Configured value: 1
-dbc DAYS_BETWEEN_DB_CLEAN, --days-between-db-clean DAYS_BETWEEN_DB_CLEAN
Set the numbers of days since the introduction of a subject into inactive-db.json for it to qualifies for deletion.
Configured value: 28
-wdb, --whois-database
Switch the value of the usage of a database to store whois data to avoid whois servers rate limit.
Configured value: True
Output control:
-a, --all Output all available information on the screen.
Configured value: True
-ex, --execution Switch the default value of the execution time showing.
Configured value: False
--hierarchical Switch the value of the hierarchical sorting of the tested file.
Configured value: False
-h, --host Switch the value of the generation of hosts file.
Configured value: True
-ip IP Change the IP to print in the hosts files with the given one.
Configured value: '0.0.0.0'
--json Switch the value of the generation of the JSON formatted list of domains.
Configured value: False
--less Output less informations on screen.
Configured value: False
-nf, --no-files Switch the value of the production of output files.
Configured value: False
-nl, --no-logs Switch the value of the production of logs files in the case we encounter some errors.
Configured value: False
-nu, --no-unified Switch the value of the production unified logs under the output directory.
Configured value: False
--percentage Switch the value of the percentage output mode.
Configured value: True
--plain Switch the value of the generation of the plain list of domains.
Configured value: False
--dots Prints dots to stdout instead of giving the impression that we hang on.
Configured value: False
-q, --quiet Run the script in quiet mode.
Configured value: False
--share-logs Switch the value of the sharing of logs.
Configured value: False
-s, --simple Switch the value of the simple output mode.
Configured value: False
--split Switch the value of the split of the generated output files.
Configured value: True
--store-whois Switch the value of the WHOIS record storage in the WHOIS DB.
Configured value: False
Multiprocessing:
-m, --multiprocess Switch the value of the usage of multiple processes.
Configured value: False
--multiprocess-merging-mode MULTIPROCESS_MERGING_MODE
Sets the multiprocess merging mode.
You can choose between the following: `live|ends`.
Configured value: 'end'
-p PROCESSES, --processes PROCESSES
Set the number of simultaneous processes to use while using multiple processes.
If omited, the number of available CPU cores will be used instead.
Configured value: 25
CI / CD:
--autosave-minutes AUTOSAVE_MINUTES
Update the minimum of minutes before we start committing to upstream under the CI mode.
Configured value: 15
--ci Switch the value of the CI mode.
Configured value: False
--ci-branch CI_BRANCH
Switch the branch name where we are going to push the temporary results.
Configured value: 'master'
--ci-distribution-branch CI_DISTRIBUTION_BRANCH
Switch the branch name where we are going to push the final results.
Configured value: 'master'
--cmd CMD Pass a command to run before each commit (except the final one) under the CI mode.
Configured value: ''
--cmd-before-end CMD_BEFORE_END
Pass a command to run before the results (final) commit under the CI mode.
Configured value: ''
--commit-autosave-message COMMIT_AUTOSAVE_MESSAGE
Replace the default autosave commit message.
Configured value: None
--commit-results-message COMMIT_RESULTS_MESSAGE
Replace the default results (final) commit message.
Configured value: None
Unique actions:
--clean Clean all files under the output directory.
--clean-all Clean all files under the output directory along with all file generated by PyFunceble.
--directory-structure
Generate the directory and files that are needed and which does not exist in the current directory.
For an in-depth usage, explanation and examples of the arguments, you should read the documentation at https://pyfunceble.readthedocs.io/en/master/
Crafted with ♥ by Nissar Chababy (@funilrys) with the help of https://pyfunceble.github.io/contributors.html && https://pyfunceble.github.io/special-thanks.html
From a GitLab CI/CD environment¶
As we offer an argument named --ci
which will
autosave in a GitLab CI/CI environment, this document try to
describe hot it works!
Configuration¶
Personal Access Token¶
A personal access token is needed in order for PyFunceble to automatically push the results.
You should get a personal GitLab access token with
the read_repository
and write_repository
scopes.
Once created and copied in a safe place, create a new masked variable
named GL_TOKEN
inside the CI/CD settings of your project.
The value of the variable should be the newly generated personal
access token.
.gitlab-ci.yml
¶
Note
This part only present a commented .gitlab-ci.yml
.
This is just an example do not take the following as
necessarly true.
You’re invited to submit changes if something stated in this document is wrong.
# Python needed, so we use the python image.
image: python:latest
variables:
# This is the Git name we have to set. (git config user.name)
GIT_EMAIL: "dead-hosts@funilrys.com"
# This is the Git Email we have to set. (git config user.email)
GIT_NAME: "GitLab CI/CD"
before_script:
# We install the development version of PyFunceble.
# If you prefer the stable version replace `pyfunceble-dev`
# with `pyfunceble`.
- pip3 install PyFunceble-dev
run:
script:
# Let's say we want our results and our PyFunceble
# infrastructure to be saved in a directory called `PyFunceble-tests`
# We move inside it.
- cd PyFunceble-tests
# We test the file `my_awesome_list` which is located inside the current directory.
# Note: we precise the `--ci` argument here,
# but you work without it if you set `ci: true` inside your `.PyFunceble.yaml`
- PyFunceble --ci -f my_awesome_list --plain
From a Travis CI container¶
As we offer an argument named --ci
to activate
the usage of PyFunceble in a Travis CI instance,
we document here what you need to know!
Configuration¶
Note
This part only present a commented .travis.yml
so that you can understand where to start.
If you need more practical examples, feel free to report to one of Dead-Hosts repositories which use PyFunceble with Travis CI.
env:
global:
# The following is your encrypted GitHub API key.
# Indeed as we are going to push to the repository, this is needed.
#- GH_TOKEN: # This can be set in the travis-ci https://travis-ci.com/repo/settings as 'Environment Variables'
# or as below: secure: encrypted code
- secure: QQdKFquFFojFT9XJ1XZp4EMoDTVoXFgqZq8XU+sCVf+pJQR6d/oKBp8rnSTCnZizWOQXUjGXUUxUpSG/dYGyBLjo3rH3rsn9ciZHVfubxbwK860w4sqibl4DvhCv2rdsFtvzXnhm4P9OL3i+krKdewh9fxpNyUU58qOgfnS7mK9FcFhb8z5ak2sxU2XRZedwm6Ro0oyVKs8kFkL4YaADfNyAHlGTfr9rVmE52WXQXQENktb9gFgR2A8ZnmLy0BCMZGkPDShJnjRDWD4DErtasLmLQvWpzOBwdbVJTY6U9KDRXVNdC9lp5E5Ba/dc0y36q6vjfgJR+QchetOtHgNbKYbLB8c26Di90OZCFJsxMNcl1Wct4qFPXkFGvjXrISW6pbdPL5Plto0Ig3iLiulhYOPVArysMIk9ymtSXP+WE7VWX01LQ1fEkIoSfeVZ2caTnCmTsoHVGRRe978CojKaT7yU45kb15hcyDrzptQ8EP2hfxeh5F7KtueQ6Rsb9LFDZMkMDKflZn6a+bRhESlmWWmYB9stzGzTurQA1E1bcSACJ8A8hG5nHBzZYJ2S+OY0PE7UdyOJ0JK0qe/67d+F9ocQdIoFpDDTdgIjHerQnD2wRg1aKPzLDb4jJTpqgr5ssPrqUAKl3st7gyaAZzCEADPDnIBDjOJS+mFWbx9DKgc=
# This is the Git name we have to set. (git config user.name)
- GIT_NAME: Travis CI
# This is the Git Email we have to set. (git config user.email)
- GIT_EMAIL: dead-hosts@funilrys.com
# This is the language we use.
language: python
# This is the python version we are going to use for the tests.
# Note: you can add any 3.x version to the list.
python:
- "3.8"
# The following will tell Travis CI to ends as fast as possible.
matrix:
fast_finish: true
# Here we are setting what Travis CI have to cache.
cache:
# We are caching pip3 as we use it to install PyFunceble
- pip3
install:
# We install the development version of PyFunceble. If you prefer the stable version replace
# `pyfunceble-dev` with `pyfunceble`.
- pip3 install pyfunceble-dev
# Our tests start here.
script:
# Let's say we want our results and our PyFunceble infrastructure to be saved in a directory
# called `PyFunceble-tests`
# We move inside it.
- cd PyFunceble-tests
# We test the file `my_awesome_list` which is located inside the current directory.
# Note: we precise the `--ci` argument here,
# but you work without it if you set `travis: true` inside your `.PyFunceble.yaml`
- PyFunceble --ci -f my_awesome_list --plain
# The following initiate email notification logic.
notifications:
# As we want to get a mail on failure and on status change, we set the following.
on_success: change
on_failure: always
Getting a GitHub token¶
For the secure
index of the .travis.yml
file, you have to generate a new GitHub token.
After you got your token, please write it or save it in a safe place as you’re going to need it every time you’re going to interact with Travis CI.
Note
The scope to set is public_repo
but you can also set others depending on your needs.
Encrypting the token for future usage under the Travis CIs’ containers¶
To encrypt the token simply replace and execute the following according to your personal case.
$ travis encrypt 'GH_TOKEN=theGeneratedToken' -r 'The content of TRAVIS_REPO_SLUG' --add
Warning
Please do not execute the following explicitly without replacing theGeneratedToken
with your previously generated GitHub token and The content of TRAVIS_REPO_SLUG
with
your repository slug.
Note
The usage of --add
ensure that the travis
program automatically add the secure
index to the .travis.yml
file.
Using the PyFunceble (Python) API¶
If you are working with a python script, module or even class, you can integrate PyFunceble to your main logic by importing it and using its API (cf: API Documentation).
This section will present some example of the way you can interact with PyFunceble from anything written in Python.
Get the availability of domains or IP¶
Note
This example can be found in our examples repository.
Todo
Add IPs in the loop.
"""
This is an example which respond to the following problematic(s):
* How can I get the avaibility of a domain or IP with PyFunceble ?
"""
# We want some coloration so we import the tool do to that :)
from PyFunceble import initiate_colorama, Fore, Style
# We import the tool to print the colored CLI logo.
from PyFunceble.cli_core import CLICore
# We import the configuration loader.
from PyFunceble import load_config
# We import the test method of the PyFunceble API.
from PyFunceble import test as PyFunceble
# We initiate the list of domains we are going to test.
DOMAINS = [
"google.com",
"tweeetttter.com",
"github.com",
"examplessss.ooooorgg",
"twitter.com",
"forest-jump"
]
# We initiate colorama.
initiate_colorama(True)
# We load our configuration.
#
# Note: We need this to print the logo but if you
# doesn't need the logo, you can ignore this.
load_config(generate_directory_structure=False)
# We print the PyFunceble logo.
CLICore.colorify_logo(home=True)
def print_result(subject, status):
"""
Given the subject and its status, we print it to STDOUT.
:param str subject: The subject we are going to print.
:param str status: The status of the domain.
"""
if status == "ACTIVE":
print(f"{Fore.GREEN + Style.BRIGHT}{domain} is {status}")
elif status == "INACTIVE":
print(f"{Fore.RED + Style.BRIGHT}{domain} is {status}")
else:
print(f"{Fore.CYAN + Style.BRIGHT}{domain} is {status}")
for domain in DOMAINS:
# We loop through the list of domain.
# And we print the domain and status with the right coloration!
print_result(domain, PyFunceble(domain))
Get the availability of URL¶
Note
This example can be found in our examples repository.
"""
This is an example which respond to the following problematic(s):
* How can I get the avaibility of an URL with PyFunceble ?
"""
# We want some coloration so we import the tool do to that :)
from PyFunceble import initiate_colorama, Fore, Style
# We import the tool to print the colored CLI logo.
from PyFunceble.cli_core import CLICore
# We import the configuration loader.
from PyFunceble import load_config
# We import the test method of the PyFunceble API.
from PyFunceble import url_test as PyFunceble
# We initiate the list of URLs we are going to test.
URLS = [
"https://google.com",
"http://tweeetttter.com",
"ftp://github.com",
"http://examplessss.ooooorgg",
"http://twitter.com",
]
# We initiate colorama.
initiate_colorama(True)
# We load our configuration.
#
# Note: We need this to print the logo but if you
# doesn't need the logo, you can ignore this.
load_config(generate_directory_structure=False)
# We print the PyFunceble logo.
CLICore.colorify_logo(home=True)
def print_result(subject, status):
"""
Given the subject and its status, we print it to STDOUT.
:param str subject: The subject we are going to print.
:param str status: The status of the domain.
"""
if status == "ACTIVE":
print(f"{Fore.GREEN + Style.BRIGHT}{domain} is {status}")
elif status == "INACTIVE":
print(f"{Fore.RED + Style.BRIGHT}{domain} is {status}")
else:
print(f"{Fore.CYAN + Style.BRIGHT}{domain} is {status}")
for url in URLS:
# We loop through the list of domain.
# And we print the domain and status with the right coloration!
print_result(url, PyFunceble(url))
Complete dataset while getting the avaibility of domains, IPs or URL¶
While using our API, you can request to see/get everything with the help of the complete=True
argument.
You’ll then get the following dict
as output.
{
"_status": None, # If some extra rules are applied, this index will keep the status before the extra rules was applied.
"_status_source": None, # If some extra rules are applied, this index will keep the source before the extra rules was applied.
"domain_syntax_validation": None, # The domain syntax validation status.
"expiration_date": None, # The expiration date of the tested subject (if found).
"http_status_code": None, # The status code of the tested subejct.
"ip4_syntax_validation": None, # The IPv4 syntax validation status.
"dns_lookup": [], # The DNS Lookup output.
"status_source": None, # The (final) source which gave us the status.
"status": None, # The (final) status returned by PyFunceble.
"tested": None, # The tested subject.
"url_syntax_validation": None, # The url syntax validation status.
"whois_record": None, # The whois record (if found).
"whois_server": None, # The whois server we use to get the whois record (if found).
}
Set custom configuration index while getting the avaibility of domains, IPs or URL¶
While using PyFunceble, you might want to set or overwritte a default behaviour.
You can do that in 2 ways. Globally or locally.
Globally¶
To set globally simply initiate the configuration loader and parse your custom configuration along with the initialization.
As example, you can do it like follow:
# We import the configuration loader.
from PyFunceble import load_config
# We set our list of indexes to overwritte.
OUR_PYFUNCEBLE_CONFIG = {"share_logs":False, "no_files": True}
# We load our configuration and parse our custom indexes.
load_config(generate_directory_structure=False, custom=OUR_PYFUNCEBLE_CONFIG)
## We can then play with PyFunceble and/or other business logic ...
Locally¶
To set globally simply parse your configuration along with the test method.
As example, you can do it like follow:
# We import the test method.
from PyFunceble import test as AvailabilityTest
# We set our list of indexes to overwritte.
OUR_PYFUNCEBLE_CONFIG = {"share_logs":False, "no_files": True}
# We get the status and parse our configuration.
status = AvailabilityTest("hello.world", config=OUR_PYFUNCEBLE_CONFIG)
## We can then manipulate the status and/or other business logic ...
Check the syntax of domains¶
Note
This example can be found in our examples repository.
"""
This is an example which respond to the following problematic(s):
* How can I check the syntax of a domain with PyFunceble ?
"""
# We want some coloration so we import the tool do to that :)
from PyFunceble import initiate_colorama, Fore, Style
# We import the tool to print the colored CLI logo.
from PyFunceble.cli_core import CLICore
# We import the configuration loader.
from PyFunceble import load_config
# We import the test method of the PyFunceble API.
from PyFunceble import is_domain as PyFunceble
# We initiate the list of domains we are going to test.
DOMAINS = [
"google.com",
"tweeetttter.com",
"github.com",
"examplessss.ooooorgg",
"twitter.com",
"forest-jump",
]
# We initiate colorama.
initiate_colorama(True)
# We load our configuration.
#
# Note: We need this to print the logo but if you
# doesn't need the logo, you can ignore this.
load_config(generate_directory_structure=False)
# We print the PyFunceble logo.
CLICore.colorify_logo(home=True)
def print_syntax_result(subject, status):
"""
Given the subject and its validation, we print it to STDOUT.
:param str subject: The subject we are going to print.
:param bool status: The validation state.
"""
if status is True:
print(f"{Fore.GREEN + Style.BRIGHT}{subject} is VALID")
else:
print(f"{Fore.CYAN + Style.BRIGHT}{subject} is INVALID")
for domain in DOMAINS:
# We loop through the list of domain.
# And we print the domain and status with the right coloration!
print_syntax_result(domain, PyFunceble(domain))
Check the syntax of IPv4s¶
Note
This example can be found in our examples repository.
"""
This is an example which respond to the following problematic(s):
* How can I check the syntax of an IPv4/IPv6 with PyFunceble ?
"""
# We want some coloration so we import the tool do to that :)
from PyFunceble import initiate_colorama, Fore, Style
# We import the tool to print the colored CLI logo.
from PyFunceble.cli_core import CLICore
# We import the configuration loader.
from PyFunceble import load_config
# We import the test method of the PyFunceble API.
from PyFunceble import is_ip as PyFunceble
# We initiate the list of IPs we are going to test.
IPS = ["216.58.207.46", "257.58.207.46"]
# We initiate colorama.
initiate_colorama(True)
# We load our configuration.
#
# Note: We need this to print the logo but if you
# doesn't need the logo, you can ignore this.
load_config(generate_directory_structure=False)
# We print the PyFunceble logo.
CLICore.colorify_logo(home=True)
def print_syntax_result(subject, status):
"""
Given the subject and its validation, we print it to STDOUT.
:param str subject: The subject we are going to print.
:param bool status: The validation state.
"""
if status is True:
print(f"{Fore.GREEN + Style.BRIGHT}{subject} is VALID")
else:
print(f"{Fore.CYAN + Style.BRIGHT}{subject} is INVALID")
for ip in IPS:
# We loop through the list of IP.
# And we print the IP and status with the right coloration!
print_syntax_result(ip, PyFunceble(ip))
Check the syntax of URLs¶
Note
This example can be found in our examples repository.
"""
This is an example which respond to the following problematic(s):
* How can I check the syntax of an URL with PyFunceble ?
"""
# We want some coloration so we import the tool do to that :)
from PyFunceble import initiate_colorama, Fore, Style
# We import the tool to print the colored CLI logo.
from PyFunceble.cli_core import CLICore
# We import the configuration loader.
from PyFunceble import load_config
# We import the test method of the PyFunceble API.
from PyFunceble import is_url as PyFunceble
# We initiate the list of URLs we are going to test.
URLS = [
"https://google.com",
"http://tweeetttter.com",
"htp://github.com",
"httpp://examplessss.ooooorgg",
"https:///twitter.com",
"http:forest-jump",
]
# We initiate colorama.
initiate_colorama(True)
# We load our configuration.
#
# Note: We need this to print the logo but if you
# doesn't need the logo, you can ignore this.
load_config(generate_directory_structure=False)
# We print the PyFunceble logo.
CLICore.colorify_logo(home=True)
def print_syntax_result(subject, status):
"""
Given the subject and its validation, we print it to STDOUT.
:param str subject: The subject we are going to print.
:param bool status: The validation state.
"""
if status is True:
print(f"{Fore.GREEN + Style.BRIGHT}{subject} is VALID")
else:
print(f"{Fore.CYAN + Style.BRIGHT}{subject} is INVALID")
for url in URLS:
# We loop through the list of URL.
# And we print the URL and status with the right coloration!
print_syntax_result(url, PyFunceble(url))