Code Documentation

Logic representation

Don’t have time to review the code and want to know exactly what PyFunceble does in general? This flowchart is there for you!

Note

This is a simplified version of what we do.

Note

The diagrams were generated with the help of yEd Graph Editor.

Domain testing

PyFunceble Logic representation for domain testing

IP testing

PyFunceble Logic representation for IP testing

URL testing

PyFunceble Logic representation for URL testing

Check

Problematic

How can we efficiently check the syntax of IP, domains, and URL?

Documentation

The tool to check the availability or syntax of domain, IP or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

Provides the syntax checker interface.

Author:
Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:
https://pyfunceble.github.io/special-thanks.html
Contributors:
https://pyfunceble.github.io/contributors.html
Project link:
https://github.com/funilrys/PyFunceble
Project documentation:
https://pyfunceble.readthedocs.io/en/dev/
Project homepage:
https://pyfunceble.github.io/

License:

Copyright 2017, 2018, 2019, 2020 Nissar Chababy

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
class PyFunceble.check.Check(subject)[source]

Provides our syntax checkers.

Parameters:subject (str) – The subject (URL, IP or domain) to check.
SPECIAL_USE_DOMAIN_NAMES_EXTENSIONS = ['onion']

Specifies the extension which are specified as “Special-Use Domain Names” and supported by our project.

Type:list
is_domain(subdomain_check=False)[source]

Checks if the given subject is a valid domain.

Parameters:subdomain_check (bool) –

Activates the subdomain checking.

Warning

Do not manually use.

Please report to is_subdomain().

Returns:The validity.
Return type:bool
is_ip()[source]

Checks if the given subject is a valid IPv4 or IPv6.

Returns:The validity.
Return type:bool
is_ip_range()[source]

Checks if the given subject is a valid IPv4 or IPv6 range.

Returns:The validity.
Return type:bool
is_ipv4()[source]

Checks if the given subject is a valid IPv4.

Returns:The validity.
Return type:bool
is_ipv4_range()[source]

Checks if the given subject is a valid IPv4 range.

Returns:The validity.
Return type:bool
is_ipv6()[source]

Checks if the given subject is a valid IPv6.

Returns:The validity.
Return type:bool
is_ipv6_range()[source]

Checks if the given subject is a valid IPv6 range.

Returns:The validity.
Return type:bool
is_reserved_ip()[source]

Checks if the given subject is a reserved IPv4 or IPv6.

Returns:The validity.
Return type:bool
is_reserved_ipv4()[source]

Checks if the given subject is a reserved IPv4.

Returns:The validity.
Return type:bool
is_reserved_ipv6()[source]

Checks if the given subject is a reserved IPv6.

Returns:The validity.
Return type:bool
is_subdomain()[source]

Checks if the given subject is a valid subdomain.

Returns:The validity.
Return type:bool
is_url(return_base=False, return_formatted=False)[source]

Checks if the given subject is a valid URL.

Parameters:
  • return_base (bool) – Allow us the return of the url base (if URL formatted correctly).
  • return_formatted (bool) – Allow us to get the formatted URL as response.
Returns:

The validity or the base if asked.

Return type:

bool, str

Exceptions

Problematic

How can we provide our very own exceptions ?

Documentation

The tool to check the availability or syntax of domain, IP or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

Provides our exceptions.

Author:
Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:
https://pyfunceble.github.io/special-thanks.html
Contributors:
https://pyfunceble.github.io/contributors.html
Project link:
https://github.com/funilrys/PyFunceble
Project documentation:
https://pyfunceble.readthedocs.io/en/dev/
Project homepage:
https://pyfunceble.github.io/

License:

Copyright 2017, 2018, 2019, 2020 Nissar Chababy

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
exception PyFunceble.exceptions.ConfigurationFileNotFound[source]

Describes a missing configuration file.

exception PyFunceble.exceptions.GitEmailNotFound[source]

Describes a missing Git Email.

exception PyFunceble.exceptions.GitHubTokenNotFound[source]

Describes a missing GitHub token.

exception PyFunceble.exceptions.GitLabTokenNotFound[source]

Describes a missing GitLab token.

exception PyFunceble.exceptions.GitNameNotFound[source]

Describes a missing Git Name.

exception PyFunceble.exceptions.NoConversionMade[source]

Describes the fact that a conversion was expected but none was made.

exception PyFunceble.exceptions.NoDownloadDestinationGiven[source]

Describes the fact that the download destination was not declared.

exception PyFunceble.exceptions.NoDownloadLinkGiven[source]

Describes the fact that no download link was declared.

exception PyFunceble.exceptions.NoExtractionMade[source]

Describes the fact that an extraction was expected but none was made.

exception PyFunceble.exceptions.NoInternetConnection[source]

Describes a missing connection.

exception PyFunceble.exceptions.PleaseUpdatePyFunceble[source]

Describes the impossiblity to continue with an older version.

exception PyFunceble.exceptions.PyFuncebleException[source]

Describes our own exceptions.

exception PyFunceble.exceptions.PyFuncebleExternalException[source]

Describes an exception which is caused by an external input.

exception PyFunceble.exceptions.PyFuncebleInternalException[source]

Describes an exception which is caused by our own logic.

exception PyFunceble.exceptions.UnableToDownload[source]

Describes a failing download.

exception PyFunceble.exceptions.UnknownSubject[source]

Describes the fact that an unknown subject is inputed.

exception PyFunceble.exceptions.UserAgentBrowserNotFound[source]

Describes the fact that we could not find a valid browser to work with.

exception PyFunceble.exceptions.UserAgentNotFound[source]

Describes the fact that we could not find a user agent to work with.

exception PyFunceble.exceptions.UserAgentPlatformNotFound[source]

Describes the fact that we could not find a valid browser to work with.

exception PyFunceble.exceptions.WrongParameterType[source]

Describes a wrong parameter type.

exception PyFunceble.exceptions.WrongParameterValue[source]

Describes a wrong parameter value.

Abstracts

Problematic

How can we work with abstracted data and information which are not directly related to our project main logic?

Documentation

The tool to check the availability or syntax of domain, IP or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

Provides the abtracted logics/data.

Author:
Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:
https://pyfunceble.github.io/special-thanks.html
Contributors:
https://pyfunceble.github.io/contributors.html
Project link:
https://github.com/funilrys/PyFunceble
Project documentation:
https://pyfunceble.readthedocs.io/en/dev/
Project homepage:
https://pyfunceble.github.io/

License:

Copyright 2017, 2018, 2019, 2020 Nissar Chababy

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

Infrastructure()

class PyFunceble.abstracts.infrastructure.Infrastructure[source]

Provides some infrastructure related abstractions.

ALEMBIC_DIRECTORY_NAME = 'alembic'

Sets the name of the alembic related directory.

Type:str
CI_MIGRATION_TRIGGER_FILE = '.trigger'

Sets the name of the file to write into for the case that we are migrating to sqlalchemy from inside a CI engine.

Type:str
CONFIGURATION_FILENAME = '.PyFunceble.yaml'

Sets the name of the user editable configuration file.

Type:str
DEFAULT_CONFIGURATION_FILENAME = '.PyFunceble_production.yaml'

Sets the name of the default configuration file.

Type:str
DOWN_FILENAME = '.pyfunceble_intern_downtime.json'

Sets the name of our own download time record file.

Type:str
ENV_FILENAME = '.pyfunceble-env'

Sets the name of our own environment file.

Type:str
HASHES_FILENAME = 'hashes_tracker.json'

Sets the name of the file which will save the hashes of the files we are testing.

IPV4_REPUTATION_FILENAME = 'ipv4_reputation.data'

Sets the name of our own copy of the IPv4 reputation data file.

Type:str

Sets the link to the production configuration file.

Type:str

Sets the link to the repository.

Type:str
USER_AGENT_FILENAME = 'user_agents.json'

Sets the name of our own copy of the user agents file.

Package()

class PyFunceble.abstracts.package.Package[source]

Provides some packaging related abstractions.

NAME = 'PyFunceble'

Sets the package name.

Type:str
VERSION = '3.3.7.dev (Teal Blauwbok: Gander)'

Sets the package version.

Type:str

Platform()

class PyFunceble.abstracts.platform.Platform[source]

Provides an easy way to get the current platform.

CYGWIN = ['cygwin', 'cygwin_nt-10.0']

Provides the list of supporter cygwin platform.

Type:lsit
MAC = ['darwin']

Provides the list of supported MAC platform.

Type:list
UNIX = ['linux', 'darwin']

Provides the list of supported unix platform.

Type:list
WINDOWS = ['windows']

Provides the list of supported windows platform.

Type:list
classmethod get()[source]

Returns the current platform.

Return type:str
classmethod is_cygwin()[source]

Checks if the current platform is in our cygwin list.

Return type:bool
classmethod is_mac_os()[source]

Checks if the current platform is in our OSX list.

Return type:bool
classmethod is_unix()[source]

Checks if the current platform is in our unix list.

Return type:bool
classmethod is_windows()[source]

Checks if the current platform is in our windows list.

Return type:bool

Version()

class PyFunceble.abstracts.package.Version[source]

Provides a simple way to compare our own versions.

classmethod compare(upstream)[source]

Compares the given versions with the local one.

Parameters:
  • local (list) – The local version converted by split_versions().
  • upstream (list) – The upstream version converted by split_versions().
Returns:

  • True: local < upstream
  • None: local == upstream
  • False: local > upstream

Return type:

bool, None

classmethod is_local_cloned()[source]

Checks if the local version is was downloaded per git clone.

classmethod is_local_dev()[source]

Checks if the local version is the development version.

classmethod literally_compare(local, upstream)[source]

Compares the given versions, literally.

Parameters:
  • local (str) – The local version converted by split_versions().
  • upstream (str) – The upstream version converted by split_versions().
Returns:

  • True: local == upstream
  • False: local != upstream

Return type:

bool

classmethod split_versions(version, return_non_digits=False)[source]

Converts the versions to a shorter one.

Parameters:
  • version (str) – The version to split.
  • return_non_digits (bool) – Activate the return of the non-digits parts of the splitted version.
Returns:

A tuple. The first index is the digit part of the version, when the second one is the the non-digit part of the version.

Return type:

tuple

CLI

Problematic

How can we easily work and distribute some CLI related logic?

Documentation

The tool to check the availability or syntax of domain, IP or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

Provides the CLI entry points.

Author:
Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:
https://pyfunceble.github.io/special-thanks.html
Contributors:
https://pyfunceble.github.io/contributors.html
Project link:
https://github.com/funilrys/PyFunceble
Project documentation:
https://pyfunceble.readthedocs.io/en/dev/
Project homepage:
https://pyfunceble.github.io/

License:

Copyright 2017, 2018, 2019, 2020 Nissar Chababy

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
PyFunceble.cli.tool()[source]

Provides the CLI.

Dispatcher()

class PyFunceble.cli.dispatcher.Dispatcher(preset, domain_or_ip=None, file_path=None, url_file_path=None, url_to_test=None, generate_results_only=False, generate_all_results_only=False)[source]

Dispatches to one of the other brain.

Note

We are talking about brain because, each PyFunceble.core.* is a unique brain which has it’s own function ;-)

Sometimes, 2 brains depends from the other one. But that’s great because working together is always great ;-)

Parameters:
  • preset (Preset) – An instance of the configuration preset.
  • domain_or_ip (list) – A list of subject to test.
  • file_path (str) –

    A file path or a link.

    Note

    If a file path is given, we read and test its content.

    Note

    If a link is given, we download it’s content and test it.

    Warning

    If given, we consider each line to be an adblock filter, a host file format or a subject to test.

  • url_file_path (str) –

    A file path or a link.

    Note

    If a file path is given, we read and test its content.

    Note

    If a link is given, we download it’s content and test it.

    Warning

    If given, we consider each line to be a URL to test.

  • url_to_test (str) – A list of URL to test.
classmethod dispatch_file_test(file_path, generate_results_only, generate_all_results_only)[source]

Dispatches to the brain in charge of file testing.

Parameters:
  • file_path (str) – The file path to test.
  • generate_results_only (bool) – Tell us to only regenerate from the data stored into the MariaDB/MySQL databases.
  • generate_all_results_only (bool) –

    Tell us to only regenerate from the data stored into the MariaDB/MySQL databases.

    Note

    The difference with generate_results_only is that it includes the retested which status didn’t changed.

dispatch_url_file_test(url_file_path, generate_results_only, generate_all_results_only)[source]

Dispatches to the brain in charge of url file testing.

Parameters:
  • url_file_path (str) – The file to test.
  • generate_results_only (bool) – Tell us to only regenerate from the data stored into the MariaDB/MySQL databases.

ExecutionTime()

class PyFunceble.cli.execution_time.ExecutionTime(action='start', last=False)[source]

Sets and returns the execution time of the program.

Parameters:
  • action (str) – The action related the execution time. Can be start or stop.
  • last (bool) – Tell the subsystem if we are at the very end of the file testing.
classmethod authorization()[source]

Checks the execution authorization.

Returns:The authorization status.
Return type:bool
classmethod calculate(start=None, end=None)[source]

Calculates the difference between starting and ending time.

Parameters:
  • start (int, str) – A starting time.
  • stop (int, str) – A ending time.
Returns:

A dict with following as index.

  • days
  • hours
  • minutes
  • seconds

as index.

Return type:

dict

format_execution_time(start=None, end=None)[source]

Formats the calculated time into a human readable format.

Parameters:
  • start (int, str) – A starting time.
  • stop (int, str) – A ending time.
Returns:

A human readable date.

Return type:

str

save(last=False)[source]

Saves the current time to our backup file.

Parameters:last (bool) – Tell us if we are at the very end of the file testing.
classmethod set_starting_time()[source]

Sets the starting time.

classmethod set_stoping_time()[source]

Sets the ending time.

Config

Problematic

  • How can we make personalization more simple?
  • How can we automatically merge the new configuration with the local one?
  • How can we link some configuration endpoints with the real logic.

Documentation

The tool to check the availability or syntax of domain, IP or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

Provides the configuration logics.

Author:
Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:
https://pyfunceble.github.io/special-thanks.html
Contributors:
https://pyfunceble.github.io/contributors.html
Project link:
https://github.com/funilrys/PyFunceble
Project documentation:
https://pyfunceble.readthedocs.io/en/dev/
Project homepage:
https://pyfunceble.github.io/

License:

Copyright 2017, 2018, 2019, 2020 Nissar Chababy

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

Load()

Merge()

Preset()

class PyFunceble.config.preset.Preset[source]

Checks or update the global configuration based on some events.

classmethod _Preset__are_we_allowed_to_overwrite(index)

Checks if we are allowed to overwrite an index.

api()[source]

Prepares the global configuration for a test from the API.

complements()[source]

Prepares the global configuration for a complements generation.

cooldown_time()[source]

Ensures that we always have a correct cooldown time.

classmethod disable(indexes)[source]

Sets the given configuration index to False.

classmethod dns_lookup_over_tcp()[source]

Ensures that the DNS lookup over tcp is properly set.

classmethod dns_nameserver()[source]

Ensures that the DNS nameserver is properly set.

classmethod enable(indexes)[source]

Sets the given configuration index to True.

file_url()[source]

Prepares the global configuration for a list of URL to test.

init_all()[source]

Initiate all presets which are independent from others.

classmethod maximal_processes()[source]

Ensures that the number of maximal processes is alway >= 2. If not, we don’t authorize the multiprocessing.

multiprocess()[source]

Prepares the global configuration for a test with multiple processes.

classmethod multiprocess_merging_mode()[source]

Ensures that a valid merging mode is given.

reputation_data()[source]

Ensures that the usage of reputation data is activated when needed.

classmethod reset_counters()[source]

Resets the counters.

simple_domain()[source]

Prepares the global configuration for a domain test.

simple_url()[source]

Prepares the global configuration for an URL test.

classmethod switch(variable, custom=False)[source]

Switches PyFunceble.CONFIGURATION variables to their opposite.

Parameters:
  • variable (str|bool) – The variable name to switch. The variable should be an index our configuration system. If we want to switch a bool variable, we should parse it here.
  • custom (bool) – Let us know if have to switch the parsed variable instead of our configuration index.
Returns:

The opposite of the configuration index or the given variable.

Return type:

bool

Raises:
Exception

When the configuration is not valid. In other words, if the PyFunceble.CONFIGURATION[variable_name] is not a bool.

syntax_test()[source]

Disables the HTTP status code if we are testing for syntax

classmethod timeout()[source]

Ensures that the timeout is always correct.

classmethod upgrade_database()[source]

Ensures that the database always have the latest state.

Converter

Problematic

How can we convert some data into something else or something our system understand.

Documentation

The tool to check the availability or syntax of domain, IP or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

Provides the converters.

Author:
Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:
https://pyfunceble.github.io/special-thanks.html
Contributors:
https://pyfunceble.github.io/contributors.html
Project link:
https://github.com/funilrys/PyFunceble
Project documentation:
https://pyfunceble.readthedocs.io/en/dev/
Project homepage:
https://pyfunceble.github.io/

License:

Copyright 2017, 2018, 2019, 2020 Nissar Chababy

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

ConverterBase()

class PyFunceble.converter.base.ConverterBase(data_to_convert)[source]

Provides the converter base.

Parameters:data_to_convert – The data to convert.
get_converted()[source]

Provides the converted data.

AdBlock()

class PyFunceble.converter.adblock.AdBlock(data_to_convert, aggressive=False)[source]

Converts an adblock filter line to a list os subject to test.

_AdBlock__decode_v1(data)

Decodes the v1.

Parameters:data (str) – A string to decode.
Return type:list
_AdBlock__decode_v2(data)

Decodes the v2.

Parameters:data (str) – A string to decode.
Return type:list
_AdBlock__decode_v3(data)

Decodes the v3.

Parameters:data (str) – A string to decode.
Return type:list
_AdBlock__decode_v4(data)

Decodes the v4.

Parameters:data (str) – A string to decode.
Return type:list
extract_base(subject)[source]

Extracts the base of the given element.

As an example:
given "hello.world/?is=beautiful" returns "hello.world"
Parameters:subject – The subject to work with.
extract_from_options(options)[source]

Exctracts the relevant data from the list of options.

Parameters:options (list) – The list of options of a rule.
Returns:The list of domains.
Return type:list
format_decoded(decoded, result=None)[source]

Formats the extracted adblock line in order to be compatible with what the system understand.

Parameters:
  • decoded (str) – The decoded data to work with.
  • result (list) – A list of the result of this method.
Returns:

The list of domains or IP compatible with the system.

Return type:

list

get_converted()[source]

Converts and return the result of the conversion.

Return type:list
classmethod ignore_it(subject)[source]

Checks if we have to ignore the given subject.

Parameters:subject (str) – The subject ot work with.
Returns:The result of the check.
Return type:bool
remove_ignored(subject)[source]

Removes the ignored element from the given list of subject.

Digit2Digits()

class PyFunceble.converter.digit2digits.Digit2Digits(data_to_convert)[source]

Converts a given digit to a 2 digits string.

File()

class PyFunceble.converter.file.File(data_to_convert)[source]

Converts a line to a subject to test.

get_converted()[source]

Provides the converted data.

Warning

This method returns return None if no subject of interest was found.

Return type:None, str, list

InternalUrl()

class PyFunceble.converter.internal_url.InternalUrl(data_to_convert)[source]

Converter of the internal URLs.

Note

The internal URLs are actually the URL that has nothing to do with what we are going to test.

They are only relevant for the software itself.

Parameters:data_to_convert (str) – The data to convert
to_right_url()[source]

Process the conversion to the right URL.

Core

Problematic

How can we get started?

Documentation

The tool to check the availability or syntax of domain, IP or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

Provides the core interfaces.

Author:
Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:
https://pyfunceble.github.io/special-thanks.html
Contributors:
https://pyfunceble.github.io/contributors.html
Project link:
https://github.com/funilrys/PyFunceble
Project documentation:
https://pyfunceble.readthedocs.io/en/dev/
Project homepage:
https://pyfunceble.github.io/

License:

Copyright 2017, 2018, 2019, 2020 Nissar Chababy

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

APICore()

class PyFunceble.core.api.APICore(subject, complete=False, configuration=None, db_file_name='api_call', is_parent=True)[source]

Provides the logic and interface for the tests from the API.

Parameters:
  • subject (str) – The element we are testing.
  • complete (bool) – Activate the return of a dictionnary with significant - if not all - data about the test.
  • configuration (dict) –

    The custom configuration to load.

    Note

    This will let you overwrite any of the previously given configuration index.

_APICore__inactive_database_management(subject, status)

Given the subject and status, we add or remove the subject from the inactive database.

availability(subject_type)[source]

Make an availability check.

Parameters:subject_type (str) –

Should be one of the following.

  • domain
  • url
domain_and_ip()[source]

Run a domain/IP availability check over the given subject.

domain_syntax()[source]

Run a domain syntax check over the given subject.

ip_range_syntax()[source]

Run an IP range syntax check over the given subject.

ip_syntax()[source]

Run an IP syntax check over the given subject.

ipv4_range_syntax()[source]

Run an IPv4 range syntax check over the given subject.

ipv4_syntax()[source]

Run an IPv4 syntax check over the given subject.

ipv6_range_syntax()[source]

Run an IPv6 range syntax check over the given subject.

ipv6_syntax()[source]

Run an IPv6 syntax check over the given subject.

reputation(subject_type)[source]

Make a reputation check.

Parameters:subject_type (str) –

Should be one of the following.

  • domain
  • url
subdomain_syntax()[source]

Run a subdomain syntax check over the given subject.

syntax(subject_type)[source]

Make a syntax check.

Parameters:subject_type (str) –

Should be one of the following.

  • domain
  • url
url()[source]

Run an URL availability check over the given subject.

url_syntax()[source]

Run an IPv4 syntax check over the given subject.

CLICore()

class PyFunceble.core.cli.CLICore[source]

Provides some methods which are dedicated for the CLI.

classmethod _CLICore__check_deprecated(upstream_version)

Checks if the local version is deprecated.

classmethod _CLICore__check_force_update(upstream_version)

Checks if we need to force the update.

If it’s the case, we will stop the tool.

classmethod _CLICore__print_messages(upstream_version)

Looks at the messages and prints the one that needs to be printed.

Print the colored logo based on global results.

Parameters:home (bool) – Tell us if we have to print the initial coloration.
classmethod compare_version_and_print_messages()[source]

Compares the local with the upstream version.

classmethod get_simple_coloration(status)[source]

Given a status we give the coloration for the simple mode.

Parameters:status (str) – An official status output.
classmethod get_up_statuses()[source]

Provides the list of up statuses.

classmethod get_upstream_version_file()[source]

Provides the upstream version file.

classmethod logs_sharing()[source]

Prints an information message when the logs sharing is activated.

classmethod print_header()[source]

Prints the header if needed.

classmethod print_nothing_to_test()[source]

Prints the nothing to test message.

classmethod save_into_database(output, filename)[source]

Saves the current status inside the database.

classmethod sort_generated_files()[source]

Sort the content of all files we generated.

classmethod stay_safe()[source]

Prints a friendly message.

FileCore()

class PyFunceble.core.file.FileCore(file, file_content_type='domain')[source]

Brain of PyFunceble for file testing.

Parameters:
  • file (str) – The file we are testing.
  • file_type (str) –

    The file type. Should be one of the following.

    • domain
    • url
_FileCore__run_single_test(subject, ignore_inactive_db_check=False)

Run a test for a single subject.

_FileCore__test_line(line, ignore_inactive_db_check=False)

Tests a given line.

cleanup(auto_continue_db, auto_save, test_completed=False)[source]

Runs the logic to run at the end of all test.

construct_and_get_shadow_file(file_stream, ignore_inactive_db_check=False)[source]

Provides a path to a file which contain the list to file.

The idea is to do a comparison between what we already tested and what we still have to test.

Downloads the file if it is an URL and return the name of the new file to test.

generate_complement_status_file(subject, status)[source]

Generate the complements status files.

generate_files(include_entries_without_changes=False)[source]

Generates all needed files.

generate_files_of_status(status, include_entries_without_changes=False)[source]

Generates the status file of all subjects of the given status.

Parameters:
  • status (str) – A status to filter.
  • include_entries_without_changes (bool) – Descriptive enough.
classmethod get_complements(auto_continue_db)[source]

Generate a list of complements to test.

classmethod get_subjects(line)[source]

Given a line, we give the list of subjects to test.

post_test_treatment(test_output, file_content_type, complements_test_started=False, auto_continue_db=None, inactive_db=None, mining=None, whois_db=None)[source]

Do the post test treatment.

run_test()[source]

Run the test of the content of the given file.

classmethod should_be_ignored(subject, auto_continue_db, inactive_db, ignore_inactive_db_check=False)[source]

Given a subject which is supposed to be tested, we check if we should ignore it.

test(subject)[source]

Tests the given subject and return it’s results.

write_in_shadow_file_if_needed(line, temp_file, ignore_inactive_db_check=False, autocontinue=None, inactive_db=None)[source]

Checks if the given line should be in the shadow file.

OurProcessWrapper()

class PyFunceble.core.multiprocess.OurProcessWrapper(*args, **kwargs)[source]

Wrapper of Process. The object of this class is just to overwrite Process.run() in order to catch exceptions.

Note

This class takes the same arguments as Process.

exception

Provides a way to check if an exception was send.

run()[source]

Overwrites Process.run().

MultiprocessCore()

class PyFunceble.core.multiprocess.MultiprocessCore(file, file_content_type='domain')[source]

Brain of PyFunceble for file testing with multiple processes.s.

Parameters:
  • file (str) – The file we are testing.
  • file_type (str) –

    The file type. Should be one of the following.

    • domain
    • url
_MultiprocessCore__check_exception(processes, manager_data)

Checks if an exception is present into the given pool of processes.

:param list processes. A list of running processes.

_MultiprocessCore__merge_processes_data(manager_data, tracker=None)

Reads all results and put them at the right location.

_MultiprocessCore__process_end_merging(finished, manager_data, tracker)

Processes the end merging.

_MultiprocessCore__process_live_merging(finished, manager_data, tracker)

Processes the live merging.

_MultiprocessCore__run_multiprocess_test(stream, manager, ignore_inactive_db_check=False, tracker=None)

Tests the content of the given file.

_MultiprocessCore__start_process(subject, manager_data, ignore_inactive_db_check=False)

Starts a new process.

construct_and_get_shadow_file(file_stream, ignore_inactive_db_check=False)[source]

Provides a path to a file which contain the list to file.

The idea is to do a comparison between what we already tested and what we still have to test.

run_test()[source]

Runs the test of the content of the given file.

test(subject, file_content_type, loader, manager_data, intern, ignore_inactive_db_check=False, custom=None)[source]

Tests the given subject and return the result.

work_process(*args)[source]

Work process for :code`construct_and_get_shadow_file`.

SimpleCore()

class PyFunceble.core.simple.SimpleCore(subject)[source]

Brain of PyFunceble for simple test.

Parameters:subject (str) – The subject we are testing.
_SimpleCore__save_in_database(dataset)

Saves the given dataset (result) into the database.

Parameters:dataset (dict, list) – The dataset to save.
domain()[source]

Handles the simple domain testing.

test(subject, subject_type)[source]

Processes a test of the given subject and return the result.

url()[source]

Handles the simple URL testing.

Database

Problematic

How can we store our most relevant information?

Documentation

The tool to check the availability or syntax of domain, IP or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

Provides the database interfaces.

Author:
Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:
https://pyfunceble.github.io/special-thanks.html
Contributors:
https://pyfunceble.github.io/contributors.html
Project link:
https://github.com/funilrys/PyFunceble
Project documentation:
https://pyfunceble.readthedocs.io/en/dev/
Project homepage:
https://pyfunceble.github.io/

License:

Copyright 2017, 2018, 2019, 2020 Nissar Chababy

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

InactiveDB()

class PyFunceble.database.inactive.InactiveDB(filename, parent_process=False)[source]

Provides the inactive database logic and interface.

Parameters:filename (str) – The name of the file we are processing.
_merge()[source]

Merges the database with the older one which has already been set into the database.

add(subject, status)[source]

Adds the given subject into the database.

Parameters:
  • subject (str) – The subject we are working with.
  • status (str) – The status of the given subject.
classmethod authorization()[source]

Provides the execution authorization.

clean()[source]

Cleans everything which is not needed anymore.

classmethod datetime()[source]

Gets the timestamp where we are going to save our current list.

Returns:The timestamp to append with the currently tested element.
Return type:int|str
get_already_tested()[source]

Returns a set of already tested subjects.

get_to_clean()[source]

Returns a set of subject to clean from the database.

get_to_retest()[source]

Returns a set of subject to restest.

initiate()[source]

Initiates the databsse.

load()[source]

Loads the content of the database file.

remove(subject)[source]

Removes all occurrences of the given subject from the database.

Parameters:subject (str) – The subject we are working with.
save()[source]

Saves the current database into the database file.

WhoisDB()

class PyFunceble.database.whois.WhoisDB(parent_process=False)[source]

Provides the WHOIS database interface and logic.

add(subject, expiration_date, server, record=None)[source]

Adds the given subject and expiration date to the database.

Parameters:
  • subject (str) – The subject we are working with.
  • expiration_date (str) – The extracted expiration date.
  • server (str) – The server we got the record from.
  • record (str) – The WHOIS record.
authorized

Provides the operation authorization.

get_expiration_date(subject)[source]

Gets the expiration date of the given subject.

Parameters:subject (str) – The subject we are working with.
Returns:The expiration date from the database.
Return type:tuple
is_time_older(subject)[source]

Checks if the expiration time of the given subject is older.

Parameters:subject (str) – The subject we are working with.

Note

Of course, we imply that the subject is in the database.

load()[source]

Loads the database file into the database.

classmethod merge(old)[source]

Merges the older version of the database into the new version.

Parameters:old (dict) – The old version of the database.
Returns:The database in the new format.
Return type:dict
save()[source]

Saves the database into the database file.

Downloader

Problematic

How can we easily download one of our static files ?

Documentation

The tool to check the availability or syntax of domain, IP or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

Provides the downloaders of all our static files.

Author:
Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:
https://pyfunceble.github.io/special-thanks.html
Contributors:
https://pyfunceble.github.io/contributors.html
Project link:
https://github.com/funilrys/PyFunceble
Project documentation:
https://pyfunceble.readthedocs.io/en/dev/
Project homepage:
https://pyfunceble.github.io/

License:

Copyright 2017, 2018, 2019, 2020 Nissar Chababy

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

DownloaderBase()

class PyFunceble.downloader.base.DownloaderBase[source]

Provides the downloader base.

DOWNTIME_INDEX = None

Used to set/track the download time of the current file.

Return type:str
REDOWNLOAD_AFTER = 1

Used to set the redownload frequency in day.

Return type:int
get_all_downtimes()[source]

Provides all download times.

Return type:dict
get_downtime()[source]

Provides the download timestamp of the current DOWN_TIME_INDEX.

Return type:dict, None
is_downtime_set()[source]

Checks if the downtime is set for the current DOWNTIME_INDEX.

Return type:bool
is_last_download_expired()[source]

Checks if the last downloaded file is expired.

process()[source]

Process the download and returns the downloaded text.

Return type:str, None
save_downtimes()[source]

Saves all downtimes.

update_downtime()[source]

Updates the current download time.

ConfigDownloader()

class PyFunceble.downloader.config.ConfigDownloader[source]

Provides the dowloader of the production configuration file.

DirectoryStructureDownloader()

class PyFunceble.downloader.directory_structure.DirectoryStructureDownloader[source]

Provides the downloader of the directory structure file.

IANADownloader()

class PyFunceble.downloader.iana.IANADownloader[source]

Provides the downloader of the iana database file.

IPv4ReputationDownloader()

class PyFunceble.downloader.ipv4_reputation.IPv4ReputationDownloader[source]

Provides the downloader of the IPv4 reputation file.

PublicSuffixDownloader()

class PyFunceble.downloader.publicsuffix.PublicSuffixDownloader[source]

Provides the downloader of the public suffix list file.

Engine

Problematic

  • How to auto continue?
  • How to auto save?
  • How to implement a proper debugger?
  • How to get subjects related to a tested subject?
  • How to work with different database types?
  • How to sort our outputs?

Documentation

The tool to check the availability or syntax of domain, IP or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

Provides the engine interfaces.

Author:
Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:
https://pyfunceble.github.io/special-thanks.html
Contributors:
https://pyfunceble.github.io/contributors.html
Project link:
https://github.com/funilrys/PyFunceble
Project documentation:
https://pyfunceble.readthedocs.io/en/dev/
Project homepage:
https://pyfunceble.github.io/

License:

Copyright 2017, 2018, 2019, 2020 Nissar Chababy

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

AutoContinue()

class PyFunceble.engine.auto_continue.AutoContinue(filename, parent_process=False)[source]

Provides the auto-continue subsystem.

_AutoContinue__generate_complements()

Generates the complements from the given list of tested.

_AutoContinue__get_or_generate_complements_json()

Gets or generates the complements while working with as JSON formatted database.

_AutoContinue__get_or_generate_complements_mysql()

Gets or generates the complements while working with as MySQL/MariaDB formatted database.

add(subject, status)[source]

Adds the given subject into the database.

classmethod authorization()[source]

Provides the execution authorization.

clean()[source]

Cleans the database.

get_already_tested()[source]

Returns the list of subjects which were already tested as a set.

get_or_generate_complements()[source]

Gets or generates the complements.

is_empty()[source]

Checks if the database related to the currently tested file is empty.

load()[source]

Loads previously saved database.

save()[source]

Saves the current state of the database.

update_counters()[source]

Updates the counters.

AutoSave()

class PyFunceble.engine.auto_save.AutoSave(start_time=None)[source]

Provides the autosave interface.

classmethod get_current_ci()[source]

Provides the current CI to use.

is_time_exceed()[source]

Checks if the end time is exceed.

process(test_completed=False)[source]

Processes the autosave base on the current state of the test.

Parameters:test_completed (bool) – Tell us if we finished the test.

Logger()

class PyFunceble.engine.logger.Logger(debug=False, on_screen=False, output_directory=None)[source]

Provides our logging logic.

_Logger__get_handler(handler_type)

Provides a handler for of the given type.

_Logger__init_loggers()

Initiates all loggers.

_Logger__set_output_directory(output_directory)

Shares the given output directory.

Note

If the given output directory does not exists, we create it.

Parameters:output_directory (string) – The output directory.
authorization(debug)[source]

Provides the operation authorization.

critical(message)[source]

Logs the critical message.

debug(message)[source]

Logs the debug message.

error(message)[source]

Logs the error message.

exception()[source]

Logs the exception message.

fatal(message)[source]

Logs the fatal message.

format_to_apply = '[%(asctime)s::%(levelname)s::%(origin_path)s:%(origin_line)s@%(origin_func)s](PID%(process)s:%(processName)s): %(message)s'

The format to apply.

classmethod get_origin_info()[source]

Returns the information about where the logger was triggered.

Returns:A tuple, which is composed of the following.

(trigger file path, trigger line, trigger function/method name)

Return type:tuple
info(message)[source]

Logs the info message.

init(output_directory=None)[source]

Initiates the logger.

root_logger_format = '[%(asctime)s::%(levelname)s](PID%(process)s:%(processName)s): %(message)s'

The format to parse to the root logger (if used).

warning(message)[source]

Logs the warning message.

Mining()

class PyFunceble.engine.mining.Mining(filename, parent_process=False)[source]

Manages the minig subsystem.

classmethod authorization()[source]

Provides the operation authorization.

classmethod get_history(url, verify=None)[source]

Gets the history of the given url.

Parameters:url (str) – An URL to call.
Returns:The list of links.
Return type:list
list_of_mined()[source]

Provides the list of mined domains so that they can be tested.

Returns:The list of mined domains.

The returned format is the following:

[
    (index_to_delete_after_test, mined),
    (index_to_delete_after_test, mined),
    (index_to_delete_after_test, mined)
]
Return type:list
load()[source]

Loads the content of the database file.

mine(subject, subject_type)[source]

Searches for domain or URL related to the original URL or domain. If some are found, we add them into the database.

Parameters:
  • subject (str) – The subject we are working with.
  • subject_typ (str) –

    The type of the subject.

    Can me one of the following:

    • url
    • domain
remove(subject, history_member)[source]

Removes the given subject from the database assigned to the currently tested file.

Parameters:
  • subject (str) – The subject we are working with.
  • history_member (str) – The history member to delete.
save()[source]

Saves the content of the database into the database file.

Sort()

class PyFunceble.engine.sort.Sort[source]

Provides some sorting presets which we can parse to PyFunceble.helpers.list.List.custom_format.

classmethod _Sort__get_url_base(element)

Provides the URL base if needed.

classmethod hierarchical(element)[source]

The idea behind this method is to sort a list of domain hierarchically.

Parameters:element (str) – The element we are currently reading.
Returns:The formatted element.
Return type:str

Note

For a domain like aaa.bbb.ccc.tdl.

A normal sorting is done in the following order:

  1. aaa
  2. bbb
  3. ccc
  4. tdl

This method allow the sorting to be done in the following order:

  1. tdl
  2. ccc
  3. bbb
  4. aaa
classmethod standard(element)[source]

Implements the standard and alphabetical sorting.

Parameters:element (str) – The element we are currently reading.
Returns:The formatted element.
Return type:str

Extractor

Problematic

How can we extract data from other data?

Documentation

The tool to check the availability or syntax of domain, IP or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

Provides the extractors.

Author:
Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:
https://pyfunceble.github.io/special-thanks.html
Contributors:
https://pyfunceble.github.io/contributors.html
Project link:
https://github.com/funilrys/PyFunceble
Project documentation:
https://pyfunceble.readthedocs.io/en/dev/
Project homepage:
https://pyfunceble.github.io/

License:

Copyright 2017, 2018, 2019, 2020 Nissar Chababy

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

ExtractorBase()

class PyFunceble.extractor.base.ExtractorBase(data)[source]

Provides the extractor base.

Parameters:data (str) – The data to work with.
get_extracted()[source]

Provides the converted data.

ExpirationDate()

class PyFunceble.extractor.expiration_date.ExpirationDate(data)[source]

Provides a way to extract the expiration date from a given WHOIS record.

Parameters:data (str) – The WHOIS record.s
_ExpirationDate__extract_it()

Try to extract the expiration date from the given data.

_ExpirationDate__format_it(data)

Formats the given data.

_ExpirationDate__format_management(index, matched)

A format management helper.

Note

Please note that the second value of the case represent the groups in order [day,month,year].

This means that a [2,1,0] will be for example for a date in format 2017-01-02 where 01 is the month.

Parameters:
  • index (int) – The identifiant of the regex.
  • matched (list) – The matched result to format.
Returns:

A list representing the expiration date. The list can be “decoded” like [day, month, year]

Return type:

list|None

Helpers

Problematic

How can we avoid writing the same thing every time?

Documentation

The tool to check the availability or syntax of domain, IP or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

Provides the helpers.

Author:
Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:
https://pyfunceble.github.io/special-thanks.html
Contributors:
https://pyfunceble.github.io/contributors.html
Project link:
https://github.com/funilrys/PyFunceble
Project documentation:
https://pyfunceble.readthedocs.io/en/dev/
Project homepage:
https://pyfunceble.github.io/

License:

Copyright 2017, 2018, 2019, 2020 Nissar Chababy

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

Command()

class PyFunceble.helpers.Command(command, encoding='utf-8')[source]

Shell command execution.

Parameters:
  • command (str) – The command to execute.
  • encoding (str) – The encoding to use to decode the shell output.
_decode_output(to_decode)[source]

Decode the output of a shell command in order to be readable.

Parameters:to_decode (bytes) – Output of a command to decode.
Returns:The decoded output.
Return type:str
execute()[source]

Execute the given command.

Returns:The output of the command.
Return type:str
run(rstrip=True)[source]

Run the given command and yield each line(s) one by one.

Note

The difference between this method and execute() is that execute() wait for the process to end in order to return its output while this method return each line one by one - as they are outputed.

Parameters:rstrip (bool) – Deactivates the rstrip of the output.
run_to_stdout()[source]

Run the given command and print each line(s) to stdout.

Dict()

class PyFunceble.helpers.Dict(main=None)[source]

Simplify some dict manipulation.

Parameters:main (dict) – The main dict to work with.
Raises:TypeError – When main is not a dict nor a list (tolarated).
classmethod from_json(json_str, return_dict_on_error=True)[source]

Converts a given JSON string to dict/list.

Parameters:return_dict_on_error (bool) – Return a dict instead of a NoneType.
Return type:dict|list
classmethod from_json_file(file_path, encoding='utf-8', return_dict_on_error=True)[source]

Reads the given file path and convert it’s content to dict/list (tolarated).

Parameters:
  • file_path (str) – The file path.
  • return_dict_on_error (bool) – Return a dict instead of a NoneType.
Return type:

dict|list

classmethod from_yaml(yaml_str)[source]

Converts the given YAML string to dict/list.

Parameters:yaml_str (str) – The YAML string to convert.
Return type:dict|list
classmethod from_yaml_file(file_path, encoding='utf-8')[source]

Converts a given YAML formatted file, into dict/list.

Parameters:
  • file_path (str) – The file path.
  • encoding (str) – The encoding to use.
Return type:

dict|list

has_same_keys_as(to_check, loop=False)[source]

A dirty solution which checks keys are presents in both given dict.

Parameters:
  • to_check (dict) – The dict to check.
  • loop (bool) – DO NOT USE, only used to tell us wen to return the list of dataset or the final result.
Return type:

bool

remove_key(key_to_remove)[source]

Remove a given key from a given dictionary.

Parameters:key_to_remove (list|str) – The key(s) to delete.
Returns:The dict without the given key(s).
Return type:dict|None
rename_key(key_to_rename, strict=True)[source]

Rename the given keys from the given dictionary.

Parameters:
  • key_to_rename (dict) –

    The key(s) to rename.

    Expected format: {old:new}

  • strict (bool) – Tell us if we have to rename the exact index or the index which looks like the given key(s)
Returns:

The well formatted dict.

Return type:

dict|None

to_json(ensure_ascii=False, indent=4, sort_keys=True)[source]

Converts a given dict to JSON and return the json string.

Parameters:
  • ensure_ascii (bool) – Avoids unicode.
  • indent (int) – The indentation to apply.
  • sort_keys – Sort the keys.
Return type:

str

to_json_file(file_path, ensure_ascii=False, indent=4, sort_keys=True, encoding='utf-8')[source]

Converts the given dict to JSON and save the result into a given file path.

Parameters:
  • file_path (str) – The file path.
  • ensure_ascii (bool) – Avoids unicode.
  • indent (int) – The indentation to apply.
  • sortkeys (bool) – Sorts the keys.
to_yaml(encoding='utf-8', default_flow_style=False, indent=4, allow_unicode=True, sort_keys=True)[source]

Converts the given dict/list to the YAML format and return the result.

Parameters:
  • encoding (str) – The encoding to use.
  • default_flow_style (bool) – Uses the default flow style.
  • indent (int) – The indentation to apply.
  • allow_unicode (bool) – Allows the decoding of unicode chars.
  • sort_keys (bool) – Sors the keys.
Return type:

dict|list

to_yaml_file(file_path, encoding='utf-8', default_flow_style=False, indent=4, allow_unicode=True, sort_keys=True)[source]

Converts the given dict/list to YAML and save the result into a file.

Parameters:
  • file_path (str) – The file path.
  • encoding (str) – The encoding.
  • default_flow_style (bool) – Uses the default flow style.
  • indent (int) – The indentation to apply.
  • allow_unicode (bool) – Allows the decoding of unicode chars.
  • sort_keys (bool) – Sorts the keys.

Directory()

class PyFunceble.helpers.Directory(dir_path=None)[source]

Simplify the directories manipulation.

:param str dir_path the path to work with.

create(dir_path=None)[source]

Creates the given directory path.

Returns:The output of self.exists after the directory creation.
Return type:bool
delete(dir_path=None)[source]

Deletes the given directory path.

Returns:not self.exists after the directory deletion.
Rtypt:bool
exists(dir_path=None)[source]

Checks if the given directory exists.

fix_path(dir_path=None, splited_path=None)[source]

Fix the path of the given path.

Note

We consider a path as fixed if it ends with the right directory separator.

classmethod get_current(with_end_sep=False)[source]

Returns the current directory path.

Parameters:with_end_sep (bool) – Add a directory separator at the end.

Download()

class PyFunceble.helpers.Download(url, verify_certificate=True, retry=3)[source]

Simplification of the downloads.

Parameters:
  • url (str) – The url to download.
  • verify_certificate (bool) – Allows/Disallows the certificate verification.
  • retry (int) – The number of time we have to retry before raising an exception.
text(destination=None)[source]

Download the body of the given url.

Note

if destination is set to None, we only return the output.

Otherwise, we save the output into the given destination, but we also return the output.

Parameters:

destination (str) – The download destination.

Return type:

str:

Raises:

File()

class PyFunceble.helpers.File(file_path=None)[source]

Simplify the file manipulations.

Parameters:file_path (str) – The file path to work with.
copy(destination)[source]

Copy the globaly given file path to the given destination.

Parameters:destination (str) – The destination of the copy.
delete(file_path=None)[source]

Deletes the given file path if it exists.

Parameters:file_path (str) –

The file path to check.

Note

If None is given, we report to the globally given file path.

Returns:The non existance state of the file.
Return type:bool
exists(file_path=None)[source]

Checks if the given file path exists.

Parameters:file_path (str) –

The file path to check.

Note

If None is given, we report to the globally given file path.

Return type:bool
get_size(file_path=None)[source]

Provides the size (in bytes) of the given file path.

Parameters:file_path (str) –

The file path to check.

Note

If None is given, we report to the globally given file path.

Return type:int
is_empty(file_path=None)[source]

Checks if the given file path is empty.

move(destination)[source]

Move the globally given file path to the given destination.

Parameters:destination (str) – The destination of the file.
read(file_path=None, encoding='utf-8')[source]

Read the given file path and return it’s content.

Parameters:
  • file_path (str) –

    The file path to check.

    Note

    If None is given, we report to the globally given file path.

  • encoding (str) – The encoding to use.
Return type:

str

write(data, overwrite=False, encoding='utf-8', file_path=None)[source]

Write the given data into the given file path.

Parameters:
  • data (str) – The data to write.
  • encoding (str) – The encoding to use while opening the file.
  • file_path (str) –

    The file path to check.

    Note

    If None is given, we report to the globally given file path.

Hash()

class PyFunceble.helpers.Hash(algo='sha512_224')[source]

Simplify the hashing of data or file content.

Parameters:algo (str) – The algorithm to use for hashing.
Raises:ValueError – When the given algo is not known.
data(data)[source]

Hash the given data.

Parameters:data (str, bytes) – The data to hash.
Return type:str
file(file_path, encoding='utf-8')[source]

Open the given file, and it’s content.

Parameters:file_path (str) – The file to hash.
Return type:str

List()

class PyFunceble.helpers.List(main, remove_empty=False)[source]

Simplify the list manipulation.

Parameters:
  • main (list) – The main list to work with.
  • remove_empty (bool) – Process the deletion of empty strings.
custom_format(key_method, reverse=False)[source]

Return a well formatted list. With the key_method as a function/method to format the elements before sorting.

Parameters:
  • key_method (function|method) – A function or method to use to format the readed element before sorting.
  • reverse (bool) – Tell us if we have to reverse the list.
Returns:

A sorted list.

Return type:

list

format()[source]

Return a well formatted list. Basicaly, it’s sort a list and remove duplicate.

Returns:A sorted, without duplicate, list.
Return type:list

Merge()

class PyFunceble.helpers.Merge(main)[source]

Simplify the merging of dict and list.

Parameters:main (str, tuple, bool, int, dict, list, float) – The main data to work with.
_Merge__dict(origin, strict=True)

Process the dict merging.

Parameters:strict (bool) – Activates the strict mode.
Return type:dict
_Merge__list(origin, strict=True)

Process the list merging.

Parameters:strict (bool) – Activates the strict mode.
Return type:list
into(origin, strict=True)[source]

Process the mergin.

Parameters:
  • origin – The original data.
  • strict (bool) – Activates the strict mode.

Regex()

class PyFunceble.helpers.Regex(regex, escape=False)[source]

Simplify the regex matching and usage.

Parameters:
  • regex (str) – The regex to use.
  • escape (bool) – Escapes the given regex.
get_matching_list(data)[source]

Returns the strings which does the match the regex in the given data.

get_not_matching_list(data)[source]

Returns the strings which does not the match the regex in the given data.

match(data, rematch=False, group=0, return_match=True)[source]

Checks if the given data match the given regex string.

Parameters:
  • data (str) – The data to work with.
  • rematch (bool) –

    The equivalent of the $BASH_REMATCH but in Python.

    It’s basically a list of all groups.

  • group (bool) – The group to return when return_match is set to True.
  • return_match (bool) – Return the part that match the given regex string.
replace_match(data, replacement, occurences=0, multiline=False)[source]

Replaces the string which match the regex string with the given replacement.

Parameters:
  • data (str) – The data to work with.
  • replacement (str) – The replacement of the matched regex.
  • occurences (int) –

    The number of occurences to replace.

    Note

    0 means all occurences.

Return type:

str

split(data)[source]

Split the reference of the given regex.

Parameters:data (str) – The data to work with.
Return type:list

Lookup

Problematic

How can we look for information about a given subject ?

Documentation

The tool to check the availability or syntax of domain, IP or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

Provides the lookup interfaces.

Author:
Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:
https://pyfunceble.github.io/special-thanks.html
Contributors:
https://pyfunceble.github.io/contributors.html
Project link:
https://github.com/funilrys/PyFunceble
Project documentation:
https://pyfunceble.readthedocs.io/en/dev/
Project homepage:
https://pyfunceble.github.io/

License:

Copyright 2017, 2018, 2019, 2020 Nissar Chababy

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

DNSLookup()

class PyFunceble.lookup.dns.DNSLookup(dns_server=None, lifetime=3, tcp=False)[source]

DNS lookup interface.

Parameters:
  • subject (str) – The subject we are working with.
  • dns_server (list|tuple|str) – The DNS server we are working with.
  • lifetime (int) – Set the lifetime of a query.
_DNSLookup__get_dns_servers_from(inputed_dns)

Given a list or an input representing dns servers, we ensure that we have a list of a string which represent an IP.

Parameters:input (str, list, tuple) – The inputed DNS server(s).
Returns:The list of dns server to use.
Return type:list
classmethod _DNSLookup__get_resolver(dns_server)

Provides the configured dns resolver.

_DNSLookup__request_complete_not_ip(subject, tcp=None)

Requests and provides the complete DNS spectrum.

Return type:dict
_DNSLookup__request_ip(subject, tcp=None)

Handle the request for a subject which is an IP.

Parameters:
  • subject (str) – The subject we are working with.
  • tcp (bool) – Tell us to use TCP for query.
Return type:

dict

Raises:

ValueError – When a non string subject is given.

_DNSLookup__request_not_ip(subject, complete=False, tcp=None)

Handle the request for a subject which is not an IP.

Return type:dict
Raises:ValueError – When a non string subject is given.
_get_server_and_port_from(inputed_dns)[source]

Given a list or an input representing dns server, we split the server from the port.

Parameters:inputed_dns – The inputed DNS server(s).
Returns:A tuple with:

(The DNS, the port)

Return type:tuple
a_record(subject, tcp=None)[source]

Return the A record of the given subject (if found).

Parameters:
  • subject (str) – The subject we are working with.
  • tcp (bool) – Tell us to use TCP for query.
Returns:

A list of A record(s).

Return type:

list, None

Raises:

ValueError – When a non string subject is given.

aaaa_record(subject, tcp=None)[source]

Return the AAAA record of the given subject (if found).

Parameters:
  • subject (str) – The subject we are working with.
  • tcp (bool) – Tell us to use TCP for query.
Returns:

A list of A record(s).

Return type:

list, None

Raises:

ValueError – When a non string subject is given.

cname_record(subject, tcp=None)[source]

Return the CNAME record of the given subject (if found).

Parameters:
  • subject (str) – The subject we are working with.
  • tcp (bool) – Tell us to use TCP for query.
Returns:

A list of CNAME record(s).

Return type:

list, None

Raises:

ValueError – When a non string subject is given.

dname_record(subject, tcp=None)[source]

Return the DNAME record of the given subject (if found).

Parameters:
  • subject (str) – The subject we are working with.
  • tcp (bool) – Tell us to use TCP for query.
Returns:

A list of DNAME record(s).

Return type:

list, None

Raises:

ValueError – When a non string subject is given.

classmethod get_addr_info(subject)[source]

Get and return the information of the given subject (address).

Parameters:subject (str) – The subject we are working with.
Returns:A list of address.
Return type:list, None
Raises:ValueError – When a non string subject is given.
classmethod get_host_by_addr(subject)[source]

Get and return the host of the given subject (address).

Parameters:subject (str) – The subject we are working with.
Returns:A dict in the following format or None.
{
    "hostname": "",
    "aliases": [],
    "ips": []
}
Return type:dict, None
Raises:ValueError – When a non string subject is given.
classmethod is_record_present_in_result(to_check, result, allow_empty=False)[source]

Checks if the given record type is in the result.

Parameters:
  • to_check (list, str, tuple) – The record to check the presence.
  • result (dict) – The result to work with.
  • allow_empty (bool) – Tell to if we allow and empty result as present.
Return type:

bool

mx_record(subject, tcp=None)[source]

Return the MX record of the given subject (if found).

Parameters:subject (str) – The subject we are working with.
Returns:A list of MX record(s).
Return type:list, None
Raises:ValueError – When a non string subject is given.
ns_record(subject, tcp=None)[source]

Return the NS record of the given subject (if found).

Parameters:
  • subject (str) – The subject we are working with.
  • tcp (bool) – Tell us to use TCP for query.
Returns:

A list of NS record(s).

Return type:

list, None

Raises:

ValueError – When a non string subject is given.

ptr_record(subject, reverse_name=True, tcp=None)[source]

Return the PTR record of the given subject (if found).

Parameters:
  • subject (str) – The subject we are working with.
  • tcp (bool) – Tell us to use TCP for query.
Returns:

A list of PTR record(s).

Return type:

list, None

Raises:

ValueError – When a non string subject is given.

request(subject, complete=False, tcp=None)[source]

Perform a DNS lookup/requests.

Parameters:
  • subject (str) – The subject we are working with.
  • complete (bool) – Tell us to return as many result as possible.
  • tcp (bool) – Tell us to use TCP for query.
Returns:

A dict with following index if the given subject is not registered into the given DNS server. (More likely local subjects).

{
    "hostname": "",
    "aliases": [],
    "ips": []
}

A dict with following index for everything else (and if found).

{
    "nameservers: [],
    "A": [],
    "AAAA": [],
    "CNAME": [],
    "DNAME": [],
    "MX": [],
    "NS": [],
    "TXT": [],
    "PTR": []
}

Return type:

dict

Raises:

ValueError – When a non string subject is given.

txt_record(subject, tcp=None)[source]

Return the TXT record of the given subject (if found).

Parameters:
  • subject (str) – The subject we are working with.
  • tcp (bool) – Tell us to use TCP for query.
Returns:

A list of TXT record(s).

Return type:

list, None

Raises:

ValueError – When a non string subject is given.

update_lifetime(lifetime)[source]

Updates the lifetime of a query.

update_nameserver(nameserver)[source]

Updates the nameserver to query.

HTTPCode()

class PyFunceble.lookup.http_code.HTTPCode(subject, subject_type)[source]

Get and return the HTTP code status of a given domain.

Parameters:
  • subject (str) – The subject we are working with.
  • _type (subject) –

    The type of the subject we are working with. Should be one of the following.

    • url
    • domain
    • file_url
    • file_domain
    • ipv6
_get_it()[source]

Get the HTTP code status.

Returns:The matched HTTP status code.
Return type:int|None
get()[source]

Return the HTTP code status.

Returns:The matched and formatted status code.
Return type:str|int|None

Iana()

class PyFunceble.lookup.iana.Iana[source]

Let us interact with the IANA database.

_Iana__get_content()

Provides the content of our IANA database.

PublicSuffix()

class PyFunceble.lookup.publicsuffix.PublicSuffix[source]

Let us interact with the public suffix database.

_PublicSuffix__get_content()

Provides the content of our PSL database.

Referer()

class PyFunceble.lookup.referer.Referer(subject)[source]

Get the WHOIS server (referer) of the current domain extension according to the IANA database.

Parameters:subject (str) – The subject we are working with.
get()[source]

Return the referer aka the WHOIS server of the current domain extension.

Returns:
  • [0] None if there is no referer.
  • [0] False if the extension is unknown which implicitly means
    that the subject is INVALID
  • [0] str The resolved IP to use.
  • [1] str, None the domain referer.
Return type:tuple

HostSSLAdapter()

class PyFunceble.lookup.requests.HostSSLAdapter(pool_connections=10, pool_maxsize=10, max_retries=0, pool_block=False)[source]

Extends the build-in HTTP Adapter for urllib3 for our needs.

_HostSSLAdapter__resolve_with_cache(hostname)

Resolve the IP of the given hostname.

Parameters:hostname (str) – The hostname to resolve.
Returns:The IP of the host name or the hostname itself.
Return type:None, str
classmethod resolve(hostname)[source]

Resolve the IP of the given hostname.

Parameters:hostname (str) – The hostname to resolve.
Returns:The IP of the host name or the hostname itself.
Return type:None, str
send(request, **kwargs)[source]

Overwrite the upstream :code:send` method.

We basically do the same. We only ensure that we request the IP from the chosen DNS record.

Parameters:
  • request – The PreparedRequest being sent.
  • stream – (optional) Whether to stream the request content.
  • timeout (float or tuple or urllib3 Timeout object) – (optional) How long to wait for the server to send data before giving up, as a float, or a (connect timeout, read timeout) tuple.
  • verify – (optional) Either a boolean, in which case it controls whether we verify the server’s TLS certificate, or a string, in which case it must be a path to a CA bundle to use
  • cert – (optional) Any user-provided SSL certificate to be trusted.
  • proxies – (optional) The proxies dictionary to apply to the request.
Return type:

requests.Response

HostAdapter()

class PyFunceble.lookup.requests.HostAdapter(pool_connections=10, pool_maxsize=10, max_retries=0, pool_block=False)[source]

Extends the build-in HTTP Adapter for urllib3 for our needs.

_HostAdapter__resolve_with_cache(hostname)

Resolve the IP of the given hostname.

Parameters:hostname (str) – The hostname to resolve.
Returns:The IP of the host name or the hostname itself.
Return type:None, str
send(request, **kwargs)[source]

Overwrite the upstream :code:send` method.

We basically do the same. We only ensure that we request the IP from the chosen DNS record.

Parameters:
  • request – The PreparedRequest being sent.
  • stream – (optional) Whether to stream the request content.
  • timeout (float or tuple or urllib3 Timeout object) – (optional) How long to wait for the server to send data before giving up, as a float, or a (connect timeout, read timeout) tuple.
  • verify – (optional) Either a boolean, in which case it controls whether we verify the server’s TLS certificate, or a string, in which case it must be a path to a CA bundle to use.
  • cert – (optional) Any user-provided SSL certificate to be trusted.
  • proxies – (optional) The proxies dictionary to apply to the request.
Return type:

requests.Response

IPv4Reputation

class PyFunceble.lookup.ipv4_reputation.IPv4Reputation[source]

Let us checks a given IPv4 against the IPv4 reputation file.

_IPv4Reputation__is_present(subject)

Checks the given subject against the input file.

Requests()

class PyFunceble.lookup.requests.Requests(max_retries=None)[source]

Handles all usage of requests.

Parameters:url (str) – The URL to work with.
delete(url, **kwargs)[source]

Sends a DELETE request. Returns Response object.

Parameters:
  • url – URL for the new Request object.
  • kwargs – Optional arguments that request takes.
Return type:

requests.Response

exceptions = <module 'requests.exceptions' from '/home/docs/checkouts/readthedocs.org/user_builds/pyfunceble/envs/dev/lib/python3.7/site-packages/requests/exceptions.py'>
get(url, **kwargs)[source]

Sends a GET request. Returns Response object.

Parameters:
  • url – URL for the new Request object.
  • kwargs – Optional arguments that request takes.
Return type:

requests.Response

head(url, **kwargs)[source]

Sends a HEAD request. Returns Response object.

Parameters:
  • url – URL for the new Request object.
  • kwargs – Optional arguments that request takes.
Return type:

requests.Response

options(url, **kwargs)[source]

Sends a OPTIONS request. Returns Response object.

Parameters:
  • url – URL for the new Request object.
  • kwargs – Optional arguments that request takes.
Return type:

requests.Response

patch(url, **kwargs)[source]

Sends a PATCH request. Returns Response object.

Parameters:
  • url – URL for the new Request object.
  • kwargs – Optional arguments that request takes.
Return type:

requests.Response

post(url, **kwargs)[source]

Sends a POST request. Returns Response object.

Parameters:
  • url – URL for the new Request object.
  • kwargs – Optional arguments that request takes.
Return type:

requests.Response

put(url, **kwargs)[source]

Sends a PUT request. Returns Response object.

Parameters:
  • url – URL for the new Request object.
  • kwargs – Optional arguments that request takes.
Return type:

requests.Response

WhoisLookup()

class PyFunceble.lookup.whois.WhoisLookup(subject, server=None, timeout=3)[source]

Implementation of the UNIX whois command.

Parameters:
  • subject (str) – The subject we are working with.
  • server (str) –

    The WHOIS server we are working with.

    Note

    If None is given, we look for the best one.

  • timeout (int) –

    The timeout to apply.

    Warning

    The timeout must be a modulo of 3.

request()[source]

Perform the WHOIS request.

Output

Problematic

How can we generate or work with outputs ?

Documentation

The tool to check the availability or syntax of domain, IP or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

Provides the output related interfaces.

Author:
Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:
https://pyfunceble.github.io/special-thanks.html
Contributors:
https://pyfunceble.github.io/contributors.html
Project link:
https://github.com/funilrys/PyFunceble
Project documentation:
https://pyfunceble.readthedocs.io/en/dev/
Project homepage:
https://pyfunceble.github.io/

License:

Copyright 2017, 2018, 2019, 2020 Nissar Chababy

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

Clean()

class PyFunceble.output.clean.Clean(clean_all=False, file_path=None)[source]

Provide the cleaning logic.

Note

By cleaning we mean the cleaning of the output directory.

Parameters:
  • list_to_test (list|None) – The list of domains we are testing.
  • clean_all (bool) – Tell the subsystem if we need to clean all. Which include, of course, the output directory but also all other file(s) generated by our system.
  • file_path (str) –

    The path to the file we tested.

    Note

    This is only relevant if you use the MariaDB/MySQL database.

almost_everything(clean_all=False, file_path=False)[source]

Delete almost all discovered files.

Parameters:clean_all (bool) – Tell the subsystem if we have to clean everything instesd of almost everything.
classmethod databases_to_delete()[source]

Set the databases files to delete.

classmethod file_to_delete(all_files=False)[source]

Return the list of file to delete.

Constructor()

class PyFunceble.output.constructor.Constructor(production=False)[source]

Basically a backup/reconstructor of our output directory.

Parameters:production (bool) – Tell the subsystem if we are preparing for production which imply the execution of the backup insteam of of the “reconstructore” mode.
classmethod _create_directory(directory, loop=False)[source]

Creates the given directory if it does not exists.

Parameters:
  • directory (str) – The directory to create.
  • loop (bool) – Tell us if we are in the creation loop or not.
_get_structure()[source]

Get the structure we are going to work with.

Returns:The structure we have to work with.
Return type:dict
_restore_replace()[source]

Check if we need to replace “.gitignore” to “.keep”.

Returns:The replacement status.
Return type:bool
_update_structure_from_config(structure)[source]

Update the paths according to configs.

Parameters:structure (dict) – The read structure.
backup()[source]

Backup the developer state of output/ in order to make it restorable and portable for user.

delete_uneeded()[source]

Delete the directory which are not registered into our structure.

restore()[source]

Restore the ‘output/’ directory structure based on the dir_structure.json file.

Generate()

class PyFunceble.output.generate.Generate(subject, subject_type, status, source=None, expiration_date=None, http_status_code=None, whois_server='Unknown', filename=None, ip_validation=False, end=False)[source]

Generate different sort of files.

Parameters:
  • subject (str) – The subject we are working with.
  • subject_type (str) – The type of the subject.
  • status (str) – The catched status.
  • source (str) – The source of the given status.
  • expiration_date (str) – The expiration date of the domain (if catched).
  • http_status_code (str|int) – The HTTP status code.
  • whois_server (str) – The whois server.
  • filename (str) – The name of the file we are testing.
  • ip_validation (bool) – The IP validation check of the currently written subject.
_Generate___get_info_files_destinations(output_hosts, output_domains, output_json)

Given the output directory, this method return several paths.

Note

The given output directories have to be partially completed.

Indeed, we only do output % final_location.

Returns:The following paths:
(
    hosts_destination,
    plain_destination,
    json_destination,
    splited_destination
)
Return type:tuple
_Generate___info_files_authorization()

Provide the authorization for the generation of info files.

Basically here is what we check:

  • We are not testing as an imported module.

and

  • The hosts file generation is activated.

or

  • The plain list generation is activated.

or

  • The “api_file_generation” was set into the CONFIGURATION.
_analytic_host_file_directory()[source]

Return the analytic directory to write depending of the matched status.

_prints_status_screen()[source]

Logic behind the printing (on screen) when generating status file.

analytic_file(new_status, old_status=None)[source]

Generate Analytic/* files based on the given old and new statuses.

Parameters:
  • new_status (str) – The new status of the domain.
  • old_status (str) – The old status of the domain.
complements_file()[source]

Generate complements files base on the current status.

file_production

Check if we are allowed to produce a file based from the given information.

Returns:The state of the production. True: We do not produce file. False: We do produce file.
Return type:bool
info_files()[source]

Generate the hosts file, the plain list, the JSON file and the splitted files.

prints_status_file()[source]

Logic behind the printing (in file) when generating status file.

status_file(exclude_file_generation=False)[source]

Generate a file according to the domain status.

Parameters:exclude_file_generation (bool) – A shorthand to disable any file generation.
unified_file()[source]

Generate unified file. Understand by that that we use an unified table instead of a separate table for each status which could result into a misunderstanding.

Logs()

class PyFunceble.output.logs.Logs(output=None)[source]

Provide a clean and unique way to work with logs. Indeed, it’s not good to have logs spread around the code :smile:

Parameters:output (str) – A path to the JSON file we are going to write.
classmethod _get_content(file)[source]

Get and return the content of the given log file.

Parameters:file (str) – The file we have to get the content from.

:return The content of the given file. :rtype: dict

classmethod _write_content(content, file)[source]

Write the content into the given file.

Parameters:
  • content (str) – The dict to write.
  • file (str) – The file to write.
expiration_date(subject, extracted)[source]

Logs the extracted expiration date.

Parameters:
  • subject (str) – The currently tested subject.
  • extracted (str) – The extracted expiration date (from WHOIS record).
referer_not_found(subject, extension)[source]

Logs the case that the referer was not found.

Parameters:
  • subject (str) – The currently tested subject.
  • extension (str) – The extension of the domain we are testing.
whois(subject, record)[source]

Logs the WHOIS record if needed.

Parameters:
  • subject (str) – The currently tested subject.
  • record (str) – The record to log.

Percentage()

class PyFunceble.output.percentage.Percentage(domain_status=None, init=None)[source]

Calculation of the percentage of each status.

Parameters:
  • domain_status (str) – The status to increment.
  • init (dict) –

    The data from a previous session we are continuing.

    Warning

    We expect the numbers and not the percentages.

classmethod calculate()[source]

Calculate the percentage of each status.

count()[source]

Count the number of domain for each status.

log()[source]

Print on screen and on file the percentages for each status.

Prints()

class PyFunceble.output.prints.Prints(to_print, template, output_file=None, only_on_file=False)[source]

Print data on screen and into a file if needed. Template Possibilities: Percentage, Less, HTTP and any status you want.

Parameters:
  • to_print (list) – The list of data to print.
  • template (str) –

    The template to use.

    Note

    Available templates:

    • Percentage
    • Less
    • HTTP
    • any of the official status.
  • output_file (str) – The path to the file to write.
  • only_on_file (bool) – Tell us if we only have to print on file and not on screen.
_Prints__get_print_size()

Provides the size of the element to print.

_colorify(data)[source]

Retun colored string.

Parameters:data (str) – The string to colorify.
Returns:A colored string.
Return type:str
_json_print()[source]

Management of the json template.

classmethod _size_from_header(header)[source]

Get the size of each columns from the header.

Parameters:header (dict) – The header template we have to get the size from.
Returns:The maximal size of the each data to print.
Return type:list
before_header()[source]

Print informations about PyFunceble and the date of generation of a file into a given path, if doesn’t exist.

data()[source]

Management and input of data to the table.

Raises:
Exception

When self.data_to_print is not a list.

data_constructor(size)[source]

Construct the table of data according to given size.

Parameters:

size (list) – The maximal length of each string in the table.

Returns:

A dict with all information about the data and how to which what maximal size to print it.

Return type:

OrderedDict

Raises:
Exception

If the data and the size does not have the same length.

header(do_not_print=False)[source]

Management and creation of templates of header. Please consider as “header” the title of each columns.

Parameters:do_not_print (bool) – Tell us if we have to print the header or not.
classmethod header_constructor(data_to_print, header_separator='-', column_separator=' ')[source]

Construct header of the table according to template.

Parameters:
  • data_to_print (list) – The list of data to print into the header of the table.
  • header_separator (str) – The separator to use between the table header and our data.
  • colomn_separator (str) – The separator to use between each colomns.
Returns:

The data to print in a list format.

Return type:

list

Status

Problematic

How can we gather, manage and provide a status for a given subject ?

Documentation

The tool to check the availability or syntax of domain, IP or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

Provides the status interfaces.

Author:
Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:
https://pyfunceble.github.io/special-thanks.html
Contributors:
https://pyfunceble.github.io/contributors.html
Project link:
https://github.com/funilrys/PyFunceble
Project documentation:
https://pyfunceble.readthedocs.io/en/dev/
Project homepage:
https://pyfunceble.github.io/

License:

Copyright 2017, 2018, 2019, 2020 Nissar Chababy

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

GathererBase()

class PyFunceble.status.gatherer_base.GathererBase(subject, filename=None, whois_db=None, inactive_db=None)[source]

Provides the gatherer base.

Parameters:
  • subject (str) – The subject to gather data for.
  • filename (str) – The filename which the subject is associated with.
  • whois_db – An instance of the whois db interface.
  • inactive_db – An instance of the inactive db interface.
gather_http_status_code()[source]

Univertialy gather the status code.

get()[source]

Provides the status.

Status()

class PyFunceble.status.status.Status(subject)[source]

The status object.

Provides the result of one of the status gatherer.

Parameters:

subject (str) – The subject which we describe.

Variables:
  • _status_source (str) – The status source before the application of our extra rules (if activated).
  • _status (str) – The status before the application of our extra rules (if activated).
  • dns_lookup (list) – The result of the DNS Lookup logic.
  • domain_syntax_validation (bool) – The domain syntax validation status.
  • expiration_date (str) – The expiration date of the subject.
  • http_status_code (str) – The HTTP status code of the subject.
  • ipv4_range_syntax_validation (bool) – The IPv4 range validation status.
  • ipv4_syntax_validation (bool) – The IPv4 syntax validation status.
  • ipv6_range_syntax_validation (bool) – The IPv6 range validation status.
  • ipv6_syntax_validation (bool) – The IPv6 syntax validation status.
  • status_source (str) – The final status source.
  • status (str) – The final status.
  • subdomain_syntax_validation (bool) – The subdomain syntax validation status.
  • tested (str) – The tested subject.
  • url_syntax_validation (bool) – The URL syntax validation status.
  • whois_server (str) – The WHOIS server we contact-ed to get the WHOIS record.
  • whois_record (str) – The WHOIS record.
get()[source]

Provides the status in a dict format.

Availability: DomainAndIp()

class PyFunceble.status.availability.domain_and_ip.DomainAndIp(subject, filename=None, whois_db=None, inactive_db=None)[source]

Gather the availability of the given IP or Domain.

_DomainAndIp__gather()

Process the gathering.

_DomainAndIp__gather_expiration_date()

Gather the expiration date.

_DomainAndIp__gather_extra_rules()

Handle the lack of WHOIS record.

Availability: ExtraRules()

class PyFunceble.status.availability.extra_rules.ExtraRules(status_obj, subject_type, http_status_code)[source]

Manage some extra rules.

Parameters:
  • status_obj (PyFunceble.status.status.Status) – The status obj to handle.
  • subject_type (str) –

    The type of the subject we are working with. Should be one of the following.

    • domain
    • url
  • http_status_code (str|int) – The extracted status code.
_ExtraRules__blogspot()

Handle the blogspot SPECIAL case.

Returns:(new status, new source) or None if there is any change to apply.
Return type:tuple|None
_ExtraRules__handle_http_code(previous_state)

Handle the HTTP Code status escalation.

Parameters:previous_state (str) – The previously catched status.
Returns:(new status, new source) or None if there is any change to apply.
Return type:tuple|None
_ExtraRules__handle_ip_range()

Handle the IP range status escalation.

Returns:(new status, new source) or None if there is any change to apply.
Return type:tuple|None
_ExtraRules__handle_potentially_inactive(previous_state)

Handle the potentially inactive case.

Parameters:previous_state (str) – The previously catched status.
Returns:(new status, new source) or None if there is any change to apply.
Return type:tuple|None
_ExtraRules__handle_potentially_up()

Handle the potentially up case.

Returns:(new status, new source) or None if there is any change to apply.
Return type:tuple|None
_ExtraRules__handle_reputation()

Handle the reputation escalation.

Returns:(new status, new source) or None if there is any change to apply.
Return type:tuple|None
classmethod _ExtraRules__http_status_code_up()

Set what we return for the HTTP Code status escalation.

Returns:(new status, new source)
Return type:tuple
classmethod _ExtraRules__special_down()

Set what we return for the SPECIAL status de-escalation.

Returns:(new status, new source)
Return type:tuple
_ExtraRules__special_down_404()

Set what we return for the SPECIAL status de-escalation when the 404 status code is caught.

Returns:(new status, new source)
Return type:tuple
_ExtraRules__special_down_410()

Set what we return for the SPECIAL status de-escalation when the 410 status code is caught.

Returns:(new status, new source)
Return type:tuple
classmethod _ExtraRules__special_up()

Set what we return for the SPECIAL status escalation.

Returns:(new status, new source)
Return type:tuple
_ExtraRules__wordpress_dot_com()

Handle the wordpress.com SPECIAL case.

Returns:(new status, new source) or None if there is any change to apply.
Return type:tuple|None
handle()[source]

Globally handle the case of the currently tested domain.

Availability: Url()

class PyFunceble.status.availability.url.Url(subject, filename=None, whois_db=None, inactive_db=None)[source]

Gather the availability of the given URL.

_Url__gather()

Process the gathering.

Reputation: DomainAndIp()

class PyFunceble.status.reputation.domain_and_ip.DomainAndIp(subject, filename=None, whois_db=None, inactive_db=None)[source]

Gather the reputation of the given domain or IP.

_DomainAndIp__gather()

Process the gathering.

Reputation: Url()

class PyFunceble.status.reputation.url.Url(subject, filename=None, whois_db=None, inactive_db=None)[source]

Gather the reputation of the given URL (base).

_Url__gather()

Process the gathering.

Syntax: DomainAndIp()

class PyFunceble.status.syntax.domain_and_ip.DomainAndIp(subject, filename=None, whois_db=None, inactive_db=None)[source]

Gather the availability of the given IP or Domain.

_DomainAndIp__gather()

Process the gathering.

Syntax: Url()

class PyFunceble.status.syntax.url.Url(subject, filename=None, whois_db=None, inactive_db=None)[source]

Gather the syntax of the given URL.

_Url__gather()

Process the gathering.