Code Documentation

Note

This section will document every part (except the API section) of our code base.

Helpers

Problematic

How can we avoid writing the same thing every time?

Documentation

The tool to check the availability or syntax of domains, IPv4 or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

This submodule will provide the helpers.

Author:
Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:
https://pyfunceble.readthedocs.io/en/master/special-thanks.html
Contributors:
http://pyfunceble.readthedocs.io/en/master/special-thanks.html
Project link:
https://github.com/funilrys/PyFunceble
Project documentation:
https://pyfunceble.readthedocs.io/en/master/
Project homepage:
https://funilrys.github.io/PyFunceble/

License:

MIT License

Copyright (c) 2017-2018 Nissar Chababy

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
class PyFunceble.helpers.Command(command)[source]

Shell command execution.

Parameters:command (str) – The command to execute
_decode_output(to_decode)[source]

Decode the output of a shell command in order to be readable.

Parameters:to_decode – Output of a command to decode.
Type:bytes
Returns:The decoded output.
Return type:str
execute()[source]

Execute the given command.

Returns:The output of the command.
Return type:str
class PyFunceble.helpers.Dict(main_dictionnary=None)[source]

Dictionary manipulations.

Parameters:main_dictionnary (dict) – The dict we are working with.
classmethod from_json(data)[source]

Convert a JSON formatted string into a dictionary.

Parameters:data (str) – A JSON formatted string to convert to dict format.
Returns:The dict representation of the JSON formatted string.
Return type:dict
classmethod from_yaml(data)[source]

Convert a YAML formatted string into a dictionary.

Parameters:data (str) – A YAML formatted string to convert to dict format.
Returns:The dict representation of the YAML formatted string.
Return type:dict
merge(to_merge, strict=True)[source]

Merge the content of to_merge into the given main dictionnary.

Parameters:
  • to_merge (dict) – The dictionnary to merge.
  • strict (bool) –

    Tell us if we have to strictly merge lists.

    True: We follow index :code`False`: We follow element (content)

Returns:

The merged dict.

Return type:

dict

remove_key(key_to_remove)[source]

Remove a given key from a given dictionary.

Parameters:key_to_remove (list|str) – The key(s) to delete.
Returns:The dict without the given key(s).
Return type:dict|None
rename_key(key_to_rename, strict=True)[source]

Rename the given keys from the given dictionary.

Parameters:
  • key_to_rename (dict) – The key(s) to rename. Expected format: {old:new}
  • strict – Tell us if we have to rename the exact index or the index which looks like the given key(s)
Returns:

The well formatted dict.

Return type:

dict|None

to_json(destination)[source]

Save a dictionnary into a JSON file.

Parameters:destination (str) – A path to a file where we’re going to write the converted dict into a JSON format.
to_yaml(destination, flow_style=False)[source]

Save a dictionnary into a YAML file.

Parameters:destination (str) – A path to a file where we’re going to write the converted dict into a JSON format.
class PyFunceble.helpers.Directory(directory)[source]

Directory manipulation.

Parameters:directory (str) – A path to the directory to manipulate.
fix_path(splited_path=None)[source]

Fix the path of the given path.

Parameters:splited_path (list) – A list to convert to the right path.
Returns:The fixed path.
Return type:str
class PyFunceble.helpers.Download(link, destination=None, return_data=False, verify_certificate=True)[source]

Download or return the content of the given link.

Parameters:
  • link (str) – The link to download.
  • destination (str) – The location where we should save the downloaded content.
  • return_data (bool) – Tell us if we need to return the page content or write its content into the given destination.
  • verify_certificate (bool) – Tell us if we need to verify the SSL/TLS certificate.
text()[source]

Download the given link and return or save its requests.text at the given destination.

Return type:

mixed

Raises:
Exception

If the status code is not 200.

class PyFunceble.helpers.File(file)[source]

File treatment/manipulations.

Parameters:file (str) – A path to the file to manipulate.
copy(destination)[source]

Copy the given file to the destination.

Parameters:destination (str) – The destination of the copy.
delete()[source]

Delete a given file path.

read()[source]

Read a given file path and return its content.

Returns:The content of the given file path.
Return type:str
write(data_to_write, overwrite=False)[source]

Write or append data into the given file path.

Parameters:
  • data_to_write (str) – The data to write.
  • overwrite (bool) – Tell us if we have to overwrite the content of the file we are working with.
class PyFunceble.helpers.Hash(file_path=None, algorithm='sha512', only_hash=False, data=None)[source]

Get and return the hash a file with the given algorithm.

Parameters:
  • file_path (str) – The path to the file we have to hash.
  • algorithm (str) – The algorithm to use.
  • only_hash – Tell us if we only have to return the desired algorithm instead of the dummy dict format.

Note

The original version can be found at https://git.io/vFQrK.

_hash_data(algo)[source]

Get hash of the given data.

Parameters:algo (str) – The algorithm to use.
_hash_file(algo)[source]

Get the hash of the given file

Parameters:algo (str) – The algorithm to use.
Returns:The hexdigest of the data.
Return type:str
get()[source]

Return the hash of the given file

class PyFunceble.helpers.List(main_list=None)[source]

List manipulation.

Parameters:main_list (list) – The list to manipulate.
custom_format(key_method, reverse=False)[source]

Return a well formatted list. With the key_method as a function/method to format the elements before sorting.

Parameters:
  • key_method (function|method) – A function or method to use to format the readed element before sorting.
  • reverse (bool) – Tell us if we have to reverse the list.
Returns:

A sorted list.

Return type:

list

format()[source]

Return a well formatted list. Basicaly, it’s sort a list and remove duplicate.

Returns:A sorted, without duplicate, list.
Return type:list
merge(to_merge, strict=True)[source]

Merge to_merge into the given main list.

Parameters:
  • to_merge (list) – The list to merge.
  • strict (bool) – Tell us if we have to respect index (True) or not (False).
Returns:

The merged list.

Return type:

list

class PyFunceble.helpers.Regex(data, regex, **args)[source]

A simple implementation ot the python.re package

Parameters:
  • data (str) – The data to check.
  • regex (str) – The regex to match.
  • group (int) – The group to return.
  • rematch (bool) –

    Allow to return the matched groups into a formatted list.

    Note

    This is an implementation of Bash ${BASH_REMATCH}

  • replace_with (str) – The value to replace the matched regex with.
  • occurences (int) – The number of occurence(s) to replace.
  • return_type – Tell us if we have to return the matched data or simply check if we matched (True) or not (False)
match()[source]

Used to get exploitable result of re.search

Returns:The data of the match status.
Return type:mixed
matching_list()[source]

Return a list of the string which match the given regex.

not_matching_list()[source]

Return a list of string which don’t match the given regex.

replace()[source]

Used to replace a matched string with another.

Returns:The data after replacement.
Return type:str

Download()

class PyFunceble.helpers.Download(link, destination=None, return_data=False, verify_certificate=True)[source]

Download or return the content of the given link.

Parameters:
  • link (str) – The link to download.
  • destination (str) – The location where we should save the downloaded content.
  • return_data (bool) – Tell us if we need to return the page content or write its content into the given destination.
  • verify_certificate (bool) – Tell us if we need to verify the SSL/TLS certificate.
text()[source]

Download the given link and return or save its requests.text at the given destination.

Return type:

mixed

Raises:
Exception

If the status code is not 200.

Command()

class PyFunceble.helpers.Command(command)[source]

Shell command execution.

Parameters:command (str) – The command to execute
_decode_output(to_decode)[source]

Decode the output of a shell command in order to be readable.

Parameters:to_decode – Output of a command to decode.
Type:bytes
Returns:The decoded output.
Return type:str
execute()[source]

Execute the given command.

Returns:The output of the command.
Return type:str

Regex()

class PyFunceble.helpers.Regex(data, regex, **args)[source]

A simple implementation ot the python.re package

Parameters:
  • data (str) – The data to check.
  • regex (str) – The regex to match.
  • group (int) – The group to return.
  • rematch (bool) –

    Allow to return the matched groups into a formatted list.

    Note

    This is an implementation of Bash ${BASH_REMATCH}

  • replace_with (str) – The value to replace the matched regex with.
  • occurences (int) – The number of occurence(s) to replace.
  • return_type – Tell us if we have to return the matched data or simply check if we matched (True) or not (False)
match()[source]

Used to get exploitable result of re.search

Returns:The data of the match status.
Return type:mixed
matching_list()[source]

Return a list of the string which match the given regex.

not_matching_list()[source]

Return a list of string which don’t match the given regex.

replace()[source]

Used to replace a matched string with another.

Returns:The data after replacement.
Return type:str

Dict()

class PyFunceble.helpers.Dict(main_dictionnary=None)[source]

Dictionary manipulations.

Parameters:main_dictionnary (dict) – The dict we are working with.
classmethod from_json(data)[source]

Convert a JSON formatted string into a dictionary.

Parameters:data (str) – A JSON formatted string to convert to dict format.
Returns:The dict representation of the JSON formatted string.
Return type:dict
classmethod from_yaml(data)[source]

Convert a YAML formatted string into a dictionary.

Parameters:data (str) – A YAML formatted string to convert to dict format.
Returns:The dict representation of the YAML formatted string.
Return type:dict
merge(to_merge, strict=True)[source]

Merge the content of to_merge into the given main dictionnary.

Parameters:
  • to_merge (dict) – The dictionnary to merge.
  • strict (bool) –

    Tell us if we have to strictly merge lists.

    True: We follow index :code`False`: We follow element (content)

Returns:

The merged dict.

Return type:

dict

remove_key(key_to_remove)[source]

Remove a given key from a given dictionary.

Parameters:key_to_remove (list|str) – The key(s) to delete.
Returns:The dict without the given key(s).
Return type:dict|None
rename_key(key_to_rename, strict=True)[source]

Rename the given keys from the given dictionary.

Parameters:
  • key_to_rename (dict) – The key(s) to rename. Expected format: {old:new}
  • strict – Tell us if we have to rename the exact index or the index which looks like the given key(s)
Returns:

The well formatted dict.

Return type:

dict|None

to_json(destination)[source]

Save a dictionnary into a JSON file.

Parameters:destination (str) – A path to a file where we’re going to write the converted dict into a JSON format.
to_yaml(destination, flow_style=False)[source]

Save a dictionnary into a YAML file.

Parameters:destination (str) – A path to a file where we’re going to write the converted dict into a JSON format.

List()

class PyFunceble.helpers.List(main_list=None)[source]

List manipulation.

Parameters:main_list (list) – The list to manipulate.
custom_format(key_method, reverse=False)[source]

Return a well formatted list. With the key_method as a function/method to format the elements before sorting.

Parameters:
  • key_method (function|method) – A function or method to use to format the readed element before sorting.
  • reverse (bool) – Tell us if we have to reverse the list.
Returns:

A sorted list.

Return type:

list

format()[source]

Return a well formatted list. Basicaly, it’s sort a list and remove duplicate.

Returns:A sorted, without duplicate, list.
Return type:list
merge(to_merge, strict=True)[source]

Merge to_merge into the given main list.

Parameters:
  • to_merge (list) – The list to merge.
  • strict (bool) – Tell us if we have to respect index (True) or not (False).
Returns:

The merged list.

Return type:

list

Directory()

class PyFunceble.helpers.Directory(directory)[source]

Directory manipulation.

Parameters:directory (str) – A path to the directory to manipulate.
fix_path(splited_path=None)[source]

Fix the path of the given path.

Parameters:splited_path (list) – A list to convert to the right path.
Returns:The fixed path.
Return type:str

File()

class PyFunceble.helpers.File(file)[source]

File treatment/manipulations.

Parameters:file (str) – A path to the file to manipulate.
copy(destination)[source]

Copy the given file to the destination.

Parameters:destination (str) – The destination of the copy.
delete()[source]

Delete a given file path.

read()[source]

Read a given file path and return its content.

Returns:The content of the given file path.
Return type:str
write(data_to_write, overwrite=False)[source]

Write or append data into the given file path.

Parameters:
  • data_to_write (str) – The data to write.
  • overwrite (bool) – Tell us if we have to overwrite the content of the file we are working with.

Hash()

class PyFunceble.helpers.Hash(file_path=None, algorithm='sha512', only_hash=False, data=None)[source]

Get and return the hash a file with the given algorithm.

Parameters:
  • file_path (str) – The path to the file we have to hash.
  • algorithm (str) – The algorithm to use.
  • only_hash – Tell us if we only have to return the desired algorithm instead of the dummy dict format.

Note

The original version can be found at https://git.io/vFQrK.

_hash_data(algo)[source]

Get hash of the given data.

Parameters:algo (str) – The algorithm to use.
_hash_file(algo)[source]

Get the hash of the given file

Parameters:algo (str) – The algorithm to use.
Returns:The hexdigest of the data.
Return type:str
get()[source]

Return the hash of the given file

AdBlock

Problematic

How can we efficiently decode AdBlock filter list?

Documentation

class PyFunceble.adblock.AdBlock(list_from_file, aggressive=False)[source]

Provide the adblock decoding logic.

Parameters:list_from_file (list) – The file in list format.
classmethod _extract_base(element)[source]

Extract the base of the given element.

Parameters:element (str|list) – The element we are working with.
classmethod _format_decoded(to_format, result=None)[source]

Format the exctracted adblock line before passing it to the system.

Parameters:
  • to_format (str) – The extracted line from the file.
  • result (list) – A list of the result of this method.
Returns:

The list of domains or IP to test.

Return type:

list

_handle_options(options)[source]

Handle the data from the options.

Parameters:options (list) – The list of options from the rule.
Returns:The list of domains to return globally.
Return type:list
classmethod _is_to_ignore(line)[source]

Check if we have to ignore the given line.

Parameters:line (str) – The line from the file.
_remove_ignored()[source]

Removed the ignored element from the given list.

decode()[source]

Decode/extract the domains to test from the adblock formated file.

Returns:The list of domains to test.
Return type:list

Auto-continue

Problematic

How can we continue the test after executable stop?

Documentation

class PyFunceble.auto_continue.AutoContinue[source]

Provide the auto-continue subsystem.

backup()[source]

Backup the current execution state.

restore()[source]

Restore data from the given path.

Auto-save

Note

Only Travis CI is actually supported.

Travis CI problematic

How can we bypass the default Travis CI timeout of 45 minutes?

Documentation

class PyFunceble.auto_save.AutoSave(is_last_domain=False, is_bypass=False)[source]

Provide the autosave logic.

Parameters:
  • is_last_domain (bool) – Tell this subsystem if we are at the very end of the file testing.
  • is_bypass (bool) – Tell this subsystem if we are in bypassing mode.
_travis()[source]

Logic behind autosave under Travis CI.

classmethod travis_permissions()[source]

Set permissions in order to avoid issues before commiting.

Check

Problematic

How can we efficiently check the format of IP, domains, and URL?

Documentation

class PyFunceble.check.Check(element=None)[source]

Provide a place to check several things around URL, IP or domain.

Parameters:element (str) – The element (URL, IP or domain) to check.
is_domain_valid(domain=None, subdomain_check=False)[source]

Check if the given domain is a valid.

Parameters:
  • domain (str) – The domain to validate.
  • subdomain_check (bool) – Activate the subdomain checking.
Returns:

The validity of the sub-domain.

Return type:

bool

is_ip_range(ip_to_check=None)[source]

Check if the given IP is a valid IPv4.

Parameters:ip_to_check (str) – The IP to test.
Returns:The validity of the IP.
Return type:bool

Note

We only test IPv4 because for now we only them for now.

is_ip_valid(ip_to_check=None)[source]

Check if the given IP is a valid IPv4.

Parameters:ip_to_check (str) – The IP to test.
Returns:The validity of the IP.
Return type:bool

Note

We only test IPv4 because for now we only them for now.

is_subdomain(domain=None)[source]

Check if the given subdomain is a subdomain.

Parameters:domain (str) – The domain to validate.
Returns:The validity of the subdomain.
Return type:bool
is_url_valid(url=None, return_base=False, return_formatted=False)[source]

Check if the given URL is valid.

Parameters:
  • url (str) – The url to validate.
  • return_base – Allow us the return of the url base (if URL formatted correctly).
  • return_formatted (bool) – Allow us to get the URL converted to IDNA if the conversion is activated.
Returns:

The validity of the URL or its base.

Return type:

bool|str

Cleaning

Problematic

How can we clean the output/ directory so we do not have a collision between old and new files?

Documentation

class PyFunceble.clean.Clean(list_to_test, clean_all=False)[source]

Provide the cleaning logic(s).

Parameters:
  • list_to_test (list|None) – The list of domains we are testing.
  • clean_all (bool) – Tell the subsystem if we need to clean all. Which include, of course, the output directory but also all other file(s) generated by our system.

Configuration

Problematics

  • How can we avoid the usage of tool.py?
  • How can we make personalization more simple?

Documentation

Load()

class PyFunceble.config.Load(path_to_config)[source]

Help us load the configuration(s) file(s).

Parameters:path_to_config (str) – The possible path to the configuration to load.
classmethod _install_directory_structure_file()[source]

Download the latest version of dir_structure_production.json.

classmethod _install_iana_config()[source]

Download iana-domains-db.json if not present.

_install_production_config()[source]

Download the production configuration and install it in the current directory.

classmethod _install_psl_config()[source]

Download public-suffix.json if not present.

_load_config_file()[source]

Load .PyFunceble.yaml into the system.

classmethod _set_path_to_configs(path_to_config)[source]

Set the paths to the configuration files.

Parameters:path_to_config (str) – The possible path to the config to load.
Returns:The path to the config to read (0), the path to the default configuration to read as fallback.(1)
Return type:tuple

Version()

class PyFunceble.config.Version(used=False)[source]

Compare the local with the upstream version.

Parameters:used (bool) – True: Version is configured for simple usage. False: Version compare local with upstream.
classmethod check_versions(local, upstream)[source]

Compare the given versions.

Parameters:
  • local (list) – The local version converted by split_versions().
  • upstream (list) – The upstream version converted by split_versions().
Returns:

  • True: local < upstream
  • None: local == upstream
  • False: local > upstream

Return type:

bool|None

classmethod check_versions_literally(local, upstream)[source]

Compare the given versions literally.

Parameters:
  • local (str) – The local version converted by split_versions().
  • upstream (str) – The upstream version converted by split_versions().
Returns:

  • True: local == upstream
  • False: local != upstream

Return type:

bool

compare()[source]

Compare the current version with the upstream saved version.

classmethod is_cloned()[source]

Let us know if we are currently in the cloned version of PyFunceble which implicitly mean that we are in developement mode.

classmethod right_url_from_version(url)[source]

Convert the GitHub URL to the right one depending of the branch or version we are working with.

Parameters:url (str) – The URL to convert.
Returns:The converted URL.
Return type:str
classmethod split_versions(version, return_non_digits=False)[source]

Convert the versions to a shorter one.

Parameters:
  • version (str) – The version to split.
  • return_non_digits (bool) – Activate the return of the non-digits parts of the splitted version.
Returns:

The splitted version name/numbers.

Return type:

list

Core

Documentation

class PyFunceble.core.Core(**args)[source]

Main entry to PyFunceble. Brain of the program. Also known as “put everything together to make the system works”.

Parameters:
  • domain_or_ip_to_test (str) – A domain or IP to test.
  • file_path (str) – A path to a file to read and test.
  • url_to_test (str) – A URL to test.
  • url_file (str) – A path to a file which contains URL to test.
  • link_to_test (str) – A link to a file to download and test.
  • modulo_test – If set to True, it will tell the system that we are working as an exported module.
  • modulo_test – bool
_entry_management()[source]

Avoid to have 1 millions line into self.__init__()

_entry_management_url()[source]

Manage the loading of the url system.

classmethod _entry_management_url_download(passed)[source]

Check if the given information is a URL. If it is the case, it download and update the location of file to test.

Parameters:passed (str) – The url passed to the system.
Returns:The state of the check.
Return type:bool
classmethod _extract_domain_from_file()[source]

Extract all non commented lines from the file we are testing.

Returns:The elements to test.
Return type:list
_file_decision(current, last, status=None)[source]

Manage the database, autosave and autocontinue systems for the case that we are reading a file.

Parameters:
  • current (str) – The currently tested element.
  • last (str) – The last element of the list.
  • status (str) – The status of the currently tested element.
_file_list_to_test_filtering()[source]

Unify the way we work before testing file contents.

classmethod _format_domain(extracted_domain)[source]

Format the extracted domain before passing it to the system.

Parameters:extracted_domain (str) – The extracted domain.
Returns:The formatted domain or IP to test.
Return type:str
classmethod _print_header()[source]

Decide if we print or not the header.

classmethod bypass()[source]

Exit the script if [PyFunceble skip] is matched into the latest commit message.

Print the colored logo based on global results.

Parameters:home (bool) – Tell us if we have to print the initial coloration.
domain(domain=None, last_domain=None)[source]

Manage the case that we want to test only a domain.

Parameters:
  • domain (str) – The domain or IP to test.
  • last_domain (str) – The last domain to test if we are testing a file.
  • return_status (bool) – Tell us if we need to return the status.
file()[source]

Manage the case that need to test each domain of a given file path.

Note

1 domain per line.

file_url()[source]

Manage the case that we have to test a file

Note

1 URL per line.

classmethod reset_counters()[source]

Reset the counters when needed.

classmethod switch(variable, custom=False)[source]

Switch PyFunceble.CONFIGURATION variables to their opposite.

Parameters:
  • variable (str|bool) – The variable name to switch. The variable should be an index our configuration system. If we want to switch a bool variable, we should parse it here.
  • custom (bool) – Let us know if have to switch the parsed variable instead of our configuration index.
Returns:

The opposite of the configuration index or the given variable.

Return type:

bool

Raises:
Exception

When the configuration is not valid. In other words, if the PyFunceble.CONFIGURATION[variable_name] is not a bool.

test(complete=False)[source]

Avoid confusion between self.domain which is called into __main__ and test() which should be called out of PyFunceble’s scope.

Parameters:

complete (bool) – Activate the return of a dictionnary with signigican data about the test.

Returns:

ACTIVE INACTIVE or INVALID.

Return type:

str|list

Raises:
Exception

When this method is called under __name__ == '__main__'

Note

This method should never be called in a __name__ == '__main__' context.

url(url_to_test=None, last_url=None)[source]

Manage the case that we want to test only a given url.

Parameters:
  • url_to_test (str) – The url to test.
  • last_url (str) – The last url of the file we are testing (if exist)

Database

Problematics

  • How can we continuously test INACTIVE and INVALID domains or IP?
  • How can we reduce the number of whois requests over time?

Documentation

class PyFunceble.database.Inactive[source]

Logic behind the generation and the usage of a database system. The main idea behind this is to provide an inactive-db.json and test all inactive domain which are into to it regularly

_add_to_test(to_add)[source]

Add an element or a list of element into PyFunceble.INTERN['inactive_db'][self.file_path]['to_test'].

Parameters:to_add (str|list) – The domain, IP or URL to add.
_backup()[source]

Save the current database into the inactive-db.json file.

_merge()[source]

Merge the real database with the older one which has already been set into PyFunceble.INTERN["inactive_db"]

_reformat_historical_formating_error()[source]

Format the old format so it can be merged into the newer format.

_retrieve()[source]

Return the current content of the inactive-db.json file.

_timestamp()[source]

Get the timestamp where we are going to save our current list.

Returns:The timestamp to append with the currently tested element.
Return type:int|str
add()[source]

Save the current :code.`PyFunceble.CONFIGURATION[‘to_test’]` into the current timestamp.

content()[source]

Get the content of the database.

Returns:The content of the database.
Return type:list
is_present()[source]

Check if the currently tested element is into the database.

remove()[source]

Remove all occurence of PyFunceble.CONFIGURATION['to_test'] from the database.

to_test()[source]

Get the list to test for the next session.

class PyFunceble.database.Whois(expiration_date=None)[source]

Logic behind the whois database. Indeed, the idea is to implement #2.

Parameters:expiration_date (str) – The extracted expiration date.
classmethod _authorization()[source]

Check if we are authorized to work with our database.

_backup()[source]

Backup the database into its file.

_retrieve()[source]

Retrieve the data from the database.

add()[source]

Add the currently tested element into the database.

get_expiration_date()[source]

Get the expiration date from the database.

Returns:The expiration date from the database.
Return type:str|None
is_in_database()[source]

Check if the element is into the database.

is_time_older()[source]

Check if the current time is older than the one in the database.

Directory Structure

Problematic

How can we give make the output directory less annoying after an update?

Documentation

class PyFunceble.directory_structure.DirectoryStructure(production=False)[source]

Basically a backup/reconstructor of our output directory.

Parameters:production (bool) – Tell the subsystem if we are preparing for production which imply the execution of the backup insteam of of the “reconstructore” mode.
classmethod _create_directory(directory, loop=False)[source]

Creates the given directory if it does not exists.

Parameters:
  • directory (str) – The directory to create.
  • loop (bool) – Tell us if we are in the creation loop or not.
_get_structure()[source]

Get the structure we are going to work with.

Returns:The structure we have to work with.
Return type:dict
_restore_replace()[source]

Check if we need to replace “.gitignore” to “.keep”.

Returns:The replacement status.
Return type:bool
_update_structure_from_config(structure)[source]

Update the paths according to configs.

Parameters:structure (dict) – The read structure.
backup()[source]
Backup the developer state of output/ in order to make it restorable
and portable for user.
delete_uneeded()[source]

Delete the directory which are not registered into our structure.

restore()[source]

Restore the ‘output/’ directory structure based on the dir_structure.json file.

Execution Time

Problematic

How to monitor the execution time of the session?

Documentation

class PyFunceble.execution_time.ExecutionTime(action='start', last=False)[source]

Set and return the exection time of the program.

Parameters:
  • action (str) – The action related the execution time. Can be start or stop.
  • last (bool) – Tell the subsystem if we are at the very end of the file testing.
classmethod _authorization()[source]

Check the execution authorization.

Returns:The authorization status.
Return type:bool
classmethod _calculate(start=None, end=None)[source]

calculate the difference between starting and ending time.

Parameters:
  • start (int|str) – A starting time.
  • stop (int|str) – A ending time.
Returns:

A dict with following as index.

  • days
  • hours
  • minutes
  • seconds

as index.

Return type:

dict

_save(last=False)[source]

Save the current time to the file.

Parameters:last (bool) – Tell us if we are at the very end of the file testing.
classmethod _starting_time()[source]

Set the starting time.

classmethod _stoping_time()[source]

Set the ending time.

format_execution_time(start=None, end=None)[source]

Format the calculated time into a human readable format.

Parameters:
  • start (int|str) – A starting time.
  • stop (int|str) – A ending time.
Returns:

A human readable date.

Return type:

str

Expiration Date

Problematic

How can we get the expiration date of a given domain?

Documentation

class PyFunceble.expiration_date.ExpirationDate[source]

Get, format and return the expiration date of a domain, if exist.

_cases_management(regex_number, matched_result)[source]

A little internal helper of self.format. (Avoiding of nested loops)

Note

Please note that the second value of the case represent the groups in order [day,month,year].

This means that a [2,1,0] will be for example for a date in format 2017-01-02 where 01 is the month.

Parameters:
  • regex_number (int) – The identifiant of the regex.
  • matched_result (list) – The matched result to format.
Returns:

A list representing the expiration date. The list can be “decoded” like [day, month, year]

Return type:

list|None

classmethod _convert_1_to_2_digits(number)[source]

Convert 1 digit number to two digits.

Parameters:number (str|int) – A number or a digit string.
Returns:A 2 or more digit string.
Return type:str
classmethod _convert_or_shorten_month(data)[source]

Convert a given month into our unified format.

Parameters:data (str) – The month to convert or shorten.
Returns:The unified month name.
Return type:str
_extract()[source]

Extract the expiration date from the whois record.

Returns:The status of the domain.
Return type:str
_format(date_to_convert=None)[source]

Format the expiration date into an unified format (01-jan-1970).

Parameters:date_to_convert (str) – The date to convert. In other words, the extracted date.
Returns:The formatted expiration date.
Return type:str
get()[source]

Execute the logic behind the meaning of ExpirationDate + return the matched status.

Returns:The status of the tested domain. Can be one of the official status.
Return type:str

Generation

Problematic

How can we generate files which reflects the results of PyFunceble?

Documentation

class PyFunceble.generate.Generate(domain_status, source=None, expiration_date=None)[source]

Generate different sort of files.

Parameters:
  • domain_status (str) – The domain status.
  • source (str) – The source of the given status.
  • expiration_date (str) – The expiration date of the domain (if catched).
_analytic_host_file_directory()[source]

Return the analytic directory to write depending of the matched status.

_do_not_produce_file()[source]

Check if we are allowed to produce a file based from the given information.

Returns:The state of the production. True: We do not produce file. False: We do produce file.
Return type:bool
classmethod _handle_non_existant_index()[source]

Handle and check that some configuration index exists.

_prints_status_file()[source]

Logic behind the printing (in file) when generating status file.

_prints_status_screen()[source]

Logic behind the printing (on screen) when generating status file.

_special_blogspot()[source]

Handle the blogspot SPECIAL case.

_special_wordpress_com()[source]

Handle the wordpress.com special case.

analytic_file(new_status, old_status)[source]

Generate Analytic/* files based on the given old and new statuses.

Parameters:
  • new_status (str) – The new status of the domain.
  • old_status (str) – The old status of the domain.
down_status_file()[source]

Logic behind the down status when generating the status file.

info_files()[source]

Generate the hosts file, the plain list and the splitted lists.

invalid_status_file()[source]

Logic behind the invalid status when generating the status file.

status_file()[source]

Generate a file according to the domain status.

unified_file()[source]

Generate unified file. Understand by that that we use an unified table instead of a separate table for each status which could result into a misunderstanding.

up_status_file()[source]

Logic behind the up status when generating the status file.

valid_status_file()[source]

Logic behind the valis status when generating the status file.

HTTP Code

Problematic

How can we get the HTTP status code of the given domain or IP?

Documentation

class PyFunceble.http_code.HTTPCode[source]

Get and return the HTTP code status of a given domain.

_access()[source]

Get the HTTP code status.

Returns:The matched HTTP status code.
Return type:int|None
get()[source]

Return the HTTP code status.

Returns:The matched and formatted status code.
Return type:str|int|None

IANA

Problematic

How can we get information from IANA?

Documentation

class PyFunceble.iana.IANA[source]

Logic behind the update and usage of iana-domains-db.json

_extensions(block)[source]

Extract the extention from the given block. Plus get its referer.

Parameters:block (str) – The line from the IANA database.
_referer(extension)[source]

Return the referer for the given extension.

Parameters:extension (str) – A valid domain extension.
Returns:The whois server to use to get the WHOIS record.
Return type:str
load()[source]

Initiate the IANA database if it is not the case.

update()[source]

Update the content of the iana-domains-db file.

Logs

Problematic

How can we efficiently generate and share logs?

Documentation

class PyFunceble.logs.Logs(output=None)[source]

Provide a clean and unique way to work with logs. Indeed, it’s not good to have logs spread around the code :smile:

Parameters:output (str) – A path to the JSON file we are going to write.
classmethod _get_content(file)[source]

Get and return the content of the given log file.

Parameters:file (str) – The file we have to get the content from.

:return The content of the given file. :rtype: dict

classmethod _write_content(content, file)[source]

Write the content into the given file.

Parameters:
  • content (dict) – The dict to write.
  • file (str) – The file to write.
expiration_date(extracted)[source]

Logs the extracted expiration date.

Parameters:extracted (str) – The extracted expiration date (from WHOIS record).
referer_not_found(extension)[source]

Logs the case that the referer was not found.

Parameters:extension (str) – The extension of the domain we are testing.
whois(record)[source]

Logs the WHOIS record if needed.

Parameters:record (str) – The record to log.

Lookup

Problematics

  • How can we get information from WHOIS records?
  • How can we check if a domain or IP have a valid pointer (nslookup)?

Documentation

class PyFunceble.lookup.Lookup[source]

Can be used to NSLOOKUP or WHOIS lookup.

classmethod nslookup()[source]

Implementation of UNIX nslookup.

classmethod whois(whois_server, domain=None, timeout=None)[source]

Implementation of UNIX whois.

Parameters:
  • whois_server (str) – The WHOIS server to use to get the record.
  • domain (str) – The domain to get the whois record from.
  • timeout (int) – The timeout to apply to the request.
Returns:

The whois record from the given whois server, if exist.

Return type:

str|None

Mining

Problematic

How can we get the list of domain or URL which link to the desired domain, IPv4 or URL?

Documentation

class PyFunceble.mining.Mining[source]

Manage the minig subsystem.

_add(to_add)[source]

Add the currently mined information to the mined “database”.

Parameters:to_add (dict) – The element to add.
_backup()[source]

Backup the mined informations.

_retrieve()[source]

Retrieve the mining informations.

list_of_mined()[source]

Provide the list of mined so they can be added to the list queue.

Returns:The list of mined domains or URL.
Return type:list
mine()[source]

Search for domain or URL related to the original URL or domain.

Returns:The mined domains or URL.
Return type:dict
process()[source]

Process the logic and structuration of the mining database.

remove()[source]

Remove the currently tested element from the mining data.

Percentage

Problematic

How can we calculate the percentage of each status?

Documentation

class PyFunceble.percentage.Percentage(domain_status=None, init=None)[source]

Calculation of the percentage of each status.

Parameters:
  • domain_status (str) – The status to increment.
  • init (dict) – The data from a previous session we are continuing.
classmethod _calculate()[source]

Calculate the percentage of each status.

count()[source]

Count the number of domain for each status.

log()[source]

Print on screen and on file the percentages for each status.

Prints

Problematic

How can we print information on the screen and on file in a table format?

Documentation

class PyFunceble.prints.Prints(to_print, template, output_file=None, only_on_file=False)[source]

Print data on screen and into a file if needed. Template Possibilities: Percentage, Less, HTTP and any status you want.

Parameters:
  • to_print – The list of data to print.
  • template (str) –

    The template to use.

    Note

    Available templates:

    • Percentage
    • Less
    • HTTP
    • any of the official status.
  • output_file (str) – The path to the file to write.
  • only_on_file (bool) – Tell us if we only have to print on file and not on screen.
_before_header()[source]

Print informations about PyFunceble and the date of generation of a file into a given path, if doesn’t exist.

_colorify(data)[source]

Retun colored string.

Parameters:data (str) – The string to colorify.
Returns:A colored string.
Return type:str
_data_constructor(size)[source]

Construct the table of data according to given size.

Parameters:

size (list) – The maximal length of each string in the table.

Returns:

A dict with all information about the data and how to which what maximal size to print it.

Return type:

OrderedDict

Raises:
Exception

If the data and the size does not have the same length.

classmethod _header_constructor(data_to_print, header_separator='-', column_separator=' ')[source]

Construct header of the table according to template.

Parameters:
  • data_to_print (list) – The list of data to print into the header of the table.
  • header_separator (str) – The separator to use between the table header and our data.
  • colomn_separator (str) – The separator to use between each colomns.
Returns:

The data to print in a list format.

Return type:

list

_json_print()[source]

Management of the json template.

classmethod _size_from_header(header)[source]

Get the size of each columns from the header.

Parameters:header (dict) – The header template we have to get the size from.
Returns:The maximal size of the each data to print.
Return type:list
data()[source]

Management and input of data to the table.

Raises:
Exception

When self.data_to_print is not a list.

header(do_not_print=False)[source]

Management and creation of templates of header. Please consider as “header” the title of each columns.

Parameters:do_not_print (bool) – Tell us if we have to print the header or not.

Production

Problematic

How can we efficiently prepare the repository for push/production?

Documentation

class PyFunceble.production.Production(extern=False)[source]

Manage and provide the production preparation logic.

Parameters:extern (bool) – Tell us if we do not have to execute the logic automatically. This allow method to be called.
classmethod is_dev_version()[source]

Check if the current branch is dev.

classmethod is_master_version()[source]

Check if the current branch is master.

Public Suffix

Problematic

How can we get the list of all possible or at least most used domain suffix?

Documentation

class PyFunceble.publicsuffix.PublicSuffix[source]

Let us interact with the public suffix database.

classmethod _data()[source]

Get the database from the public suffix repository.

_extensions(line)[source]

Extract the extension from the given line.

Parameters:line (str) – The line from the official public suffix repository.
load()[source]

Load the public suffix database into the system.

update()[source]

Update of the content of the public-suffix.json.

Referer

Problematic

How can we efficiently get the whois server to call for whois record request?

Documentation

class PyFunceble.referer.Referer[source]

Get the WHOIS server (referer) of the current domain extension according to the IANA database.

get()[source]

Return the referer aka the WHOIS server of the current domain extension.

Sort

Problematic

How can we format the list to test (and the outputted information) in a format other than the alphabetical format?

Documentation

class PyFunceble.sort.Sort[source]

Provide some sorting presets which we can parse to PyFunceble.helpers.List().custom_format().

classmethod hierarchical(element)[source]

The idea behind this method is to sort a list of domain hierarchicaly.

Parameters:element (str) – The element we are currently reading.
Returns:The formatted element.
Return type:str

Note

For a domain like aaa.bbb.ccc.tdl.

A normal sorting is done in the following order:
  1. aaa
  2. bbb
  3. ccc
  4. tdl
This method allow the sorting to be done in the following order:
  1. tdl
  2. ccc
  3. bbb
  4. aaa
classmethod standard(element)[source]

Implement the standard and alphabetical sorting.

Parameters:element (str) – The element we are currently reading.
Returns:The formatted element.
Return type:str

Status

Problematic

How can we efficiently manage the statuses in function of the test type?

Documentation

Normal testing

class PyFunceble.status.Status(matched_status, invalid_source='IANA')[source]

Hanle the research of domain status in case we don’t use WHOIS or in case that WHOIS record is not readable nor exploitable.

Parameters:matched_result (str) – The previously catched status.
handle()[source]

Handle the lack of WHOIS. :smile_cat:

Returns:The strus of the domain after generating the files desired by the user.
Return type:str

URL testing

class PyFunceble.status.URLStatus(catched_status)[source]

Generate everything around the catched status when testing for URL.

Parameters:catched_status (str) – THe catched status.
handle()[source]

Handle the backend of the given status.

Syntax checking

class PyFunceble.status.SyntaxStatus(catched_status)[source]

Generate everything around the catched status when testing for Syntax.

Parameters:catched_status (str) – THe catched status.
handle()[source]

Handle the backend of the given status.

Syntax Checking

Problematic

How can we check for syntax directly from the CLI?

Documentation

class PyFunceble.syntax.Syntax[source]

Manage everything around the Syntax testing.

classmethod get()[source]

Execute the logic behind the Syntax handling.

Returns:The syntax status.
Return type:str

URL Testing

Problematic

How can we test full URL?

Documentation

class PyFunceble.url.URL[source]

Manage everything around the URL testing.

classmethod get()[source]

Execute the logic behind the URL handling.

Returns:The status of the URL.
Return type:str