PyFunceble.cli.processes.workers package¶

Submodules¶

PyFunceble.cli.processes.workers.base module¶

The tool to check the availability or syntax of domain, IP or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

Provides the base of all our workers.

Author:: Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:: https://pyfunceble.github.io/#/special-thanks
Contributors:: https://pyfunceble.github.io/#/contributors
Project link:: https://github.com/funilrys/PyFunceble
Project documentation:: https://pyfunceble.readthedocs.io/en/dev/
Project homepage:: https://pyfunceble.github.io/

License:

Copyright 2017, 2018, 2019, 2020 Nissar Chababy

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

class PyFunceble.cli.processes.workers.base.WorkerBase(input_queue: Optional[queue.Queue], output_queue: Optional[queue.Queue] = None, global_exit_event: Optional[multiprocessing.context.BaseContext.Event] = None, *, name: Optional[str] = None, daemon: Optional[bool] = None, continuous_integration: Optional[PyFunceble.cli.continuous_integration.base.ContinuousIntegrationBase] = None, configuration: Optional[dict] = None)[source]¶

Bases: multiprocessing.context.Process

Provides the base of all our workers.

Parameters:	input_queue – The input queue to read. output_queue – The output queue to write.

BREAKOFF = 0.2¶

MINING_WAIT_TIME = 10¶

STD_NAME = 'pyfunceble_base_worker'¶

_child_connection = None¶

_exception = None¶

_parent_connection = None¶

accept_waiting_delay = None¶

add_to_input_queue(data: Any, *, worker_name: Optional[str] = None) → PyFunceble.cli.processes.workers.base.WorkerBase[source]¶

Adds the given data to the current queue.

Parameters:	data – The data to add into the queue.

add_to_output_queue(data: Any, *, worker_name: Optional[str] = None) → PyFunceble.cli.processes.workers.base.WorkerBase[source]¶

Adds the given data to the output queue queue.

Parameters:	data – The data to add into the queue.

continuous_integration = None¶

exception¶: Provides the exception of the current worker.

exit_it = None¶

global_exit_event = None¶

input_queue = None¶

output_queue = None¶

run() → None[source]¶: Method to be run in sub-process; can be overridden in sub-class

send_feeding_message = None¶

target(consumed: Any) → Optional[Tuple[Any, ...]][source]¶: This the target that is run to process something. This method should return a result which will pu send to the output queue.

terminate() → None[source]¶: Terminate our runner.

PyFunceble.cli.processes.workers.file_sorter module¶

The tool to check the availability or syntax of domain, IP or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

Provides our file sorter worker. This is the description of a file sorter worker.

Author:: Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:: https://pyfunceble.github.io/#/special-thanks
Contributors:: https://pyfunceble.github.io/#/contributors
Project link:: https://github.com/funilrys/PyFunceble
Project documentation:: https://pyfunceble.readthedocs.io/en/dev/
Project homepage:: https://pyfunceble.github.io/

License:

Copyright 2017, 2018, 2019, 2020 Nissar Chababy

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

class PyFunceble.cli.processes.workers.file_sorter.FileSorterWorker(input_queue: Optional[queue.Queue], output_queue: Optional[queue.Queue] = None, global_exit_event: Optional[multiprocessing.context.BaseContext.Event] = None, *, name: Optional[str] = None, daemon: Optional[bool] = None, continuous_integration: Optional[PyFunceble.cli.continuous_integration.base.ContinuousIntegrationBase] = None, configuration: Optional[dict] = None)[source]¶

Bases: PyFunceble.cli.processes.workers.base.WorkerBase

Provides our file sorter worker. The objective of this worker is to provides a single worker (or process if you prefer) which will be used to handle the sorting of the content of all generated files.

MAX_LINES = 1000¶

STD_NAME = 'pyfunceble_file_sorter_worker'¶

static get_files_to_sort(output_dir: str) → List[str][source]¶

Provides the list of files to sort.

Parameters:	output_dir – The directory to start from.

process_file_sorting(file: str) → None[source]¶

Process the sorting of the given file.

The idea is to split the file piece by piece and at the end join all sorted files. For that job, we create a temporary directory which will store the temporary files.

target(consumed: Any) → Optional[Tuple[Any, ...]][source]¶: This the target that is run to process something. This method should return a result which will pu send to the output queue.

PyFunceble.cli.processes.workers.migrator module¶

The tool to check the availability or syntax of domain, IP or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

Provides our migrator (base) worker.

Author:: Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:: https://pyfunceble.github.io/#/special-thanks
Contributors:: https://pyfunceble.github.io/#/contributors
Project link:: https://github.com/funilrys/PyFunceble
Project documentation:: https://pyfunceble.readthedocs.io/en/dev/
Project homepage:: https://pyfunceble.github.io/

License:

Copyright 2017, 2018, 2019, 2020 Nissar Chababy

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

class PyFunceble.cli.processes.workers.migrator.MigratorWorker(input_queue: Optional[queue.Queue], output_queue: Optional[queue.Queue] = None, global_exit_event: Optional[multiprocessing.context.BaseContext.Event] = None, *, name: Optional[str] = None, daemon: Optional[bool] = None, continuous_integration: Optional[PyFunceble.cli.continuous_integration.base.ContinuousIntegrationBase] = None, configuration: Optional[dict] = None)[source]¶

Bases: PyFunceble.cli.processes.workers.base.WorkerBase

Provides our migrator (base) worker. The objective of this worker is to provides a single worker (or process if you prefer) which will be used to handle the mining of dataset to test.

STD_NAME = 'pyfunceble_migrator_worker'¶

run() → None[source]¶: Method to be run in sub-process; can be overridden in sub-class

PyFunceble.cli.processes.workers.miner module¶

The tool to check the availability or syntax of domain, IP or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

Provides our miner worker. This is the description of a single miner worker.

Author:: Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:: https://pyfunceble.github.io/#/special-thanks
Contributors:: https://pyfunceble.github.io/#/contributors
Project link:: https://github.com/funilrys/PyFunceble
Project documentation:: https://pyfunceble.readthedocs.io/en/dev/
Project homepage:: https://pyfunceble.github.io/

License:

Copyright 2017, 2018, 2019, 2020 Nissar Chababy

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

class PyFunceble.cli.processes.workers.miner.MinerWorker(input_queue: Optional[queue.Queue], output_queue: Optional[queue.Queue] = None, global_exit_event: Optional[multiprocessing.context.BaseContext.Event] = None, *, name: Optional[str] = None, daemon: Optional[bool] = None, continuous_integration: Optional[PyFunceble.cli.continuous_integration.base.ContinuousIntegrationBase] = None, configuration: Optional[dict] = None)[source]¶

Bases: PyFunceble.cli.processes.workers.base.WorkerBase

Provides our miner worker. The objective of this worker is to provides a single worker (or process if you prefer) which will be used to handle the mining of dataset to test.

INACTIVE_STATUSES = ('INACTIVE', 'INVALID')¶

STD_NAME = 'pyfunceble_miner_worker'¶

static mine_from(subject: str) → Optional[List[str]][source]¶

Given the subject to work from, try to get the related subjects.

Parameters:	subject – The URL to start from.

target(consumed: Tuple[dict, PyFunceble.checker.status_base.CheckerStatusBase]) → None[source]¶: This the target that is run to process something. This method should return a result which will pu send to the output queue.

url2netloc = None¶

PyFunceble.cli.processes.workers.producer module¶

The tool to check the availability or syntax of domain, IP or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

Provides our producer worker. This is the description of a single producer worker.

Author:: Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:: https://pyfunceble.github.io/#/special-thanks
Contributors:: https://pyfunceble.github.io/#/contributors
Project link:: https://github.com/funilrys/PyFunceble
Project documentation:: https://pyfunceble.readthedocs.io/en/dev/
Project homepage:: https://pyfunceble.github.io/

License:

Copyright 2017, 2018, 2019, 2020 Nissar Chababy

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

class PyFunceble.cli.processes.workers.producer.ProducerWorker(input_queue: Optional[queue.Queue], output_queue: Optional[queue.Queue] = None, global_exit_event: Optional[multiprocessing.context.BaseContext.Event] = None, *, name: Optional[str] = None, daemon: Optional[bool] = None, continuous_integration: Optional[PyFunceble.cli.continuous_integration.base.ContinuousIntegrationBase] = None, configuration: Optional[dict] = None)[source]¶

Bases: PyFunceble.cli.processes.workers.base.WorkerBase

Provides our producer worker. The objective of this worker is to provides a single worker (or process if you prefer) which will be used to handle the production of output to stdout or files.

INACTIVE_STATUSES = ('INACTIVE', 'INVALID')¶

STD_NAME = 'pyfunceble_producer_worker'¶

continue_dataset = None¶

counter = None¶

file_printer = None¶

header_already_printed = None¶

inactive_dataset = None¶

run_continue_backup(test_dataset: dict, test_result: PyFunceble.checker.status_base.CheckerStatusBase) → None[source]¶: Runs the backup or update of the auto-continue dataset storage.

run_counter(test_dataset: dict, test_result: PyFunceble.checker.status_base.CheckerStatusBase) → None[source]¶: Runs the counter of the current file.

run_ignored_file_printer(test_dataset: dict, test_result: str) → None[source]¶: Runs the analytic behind the file printer.

Warning

Thie method assume that the givne dataset is ignored from the normal file printer.

run_inactive_backup(test_dataset: dict, test_result: PyFunceble.checker.status_base.CheckerStatusBase) → None[source]¶

Runs the backup or update of the Inactive dataset storage.

The idea is that if the status is OK (active), we just remove it from the dataset storage. Otherwise, we just keep it in there :-)

run_status_file_printer(test_dataset: dict, test_result: PyFunceble.checker.status_base.CheckerStatusBase) → None[source]¶: Runs the status file printer.

run_stdout_printer(test_result: PyFunceble.checker.status_base.CheckerStatusBase) → None[source]¶

Runs the stdout printer (if necessary).

Parameters:	test_result – The rest result dataset.

run_whois_backup(test_result: PyFunceble.checker.status_base.CheckerStatusBase) → None[source]¶

Runs the backup or update of the WHOIS record in our dataset storage.

Parameters:	test_result – The test result.

should_we_block_status_file_printer(test_dataset: dict, test_result: PyFunceble.checker.status_base.CheckerStatusBase) → bool[source]¶

Checks if we should block the file printer.

The reason behindn this is that we don’t want to generate an output when a subject was already into the inactive database.

static should_we_ignore(test_result: PyFunceble.checker.status_base.CheckerStatusBase) → bool[source]¶

Checks if we should ignore the given datasets.

Parameters:	test_result – The test result to check.

static should_we_print_status_to_stdout(status: str) → bool[source]¶

Checks if we are allows to print the given status (to stdout).

Parameters:	status – The status to check.

status_file_generator = None¶

stdout_printer = None¶

target(consumed: Any) → Optional[Tuple[Any, ...]][source]¶: This the target that is run to process something. This method should return a result which will pu send to the output queue.

whois_dataset = None¶

PyFunceble.cli.processes.workers.tester module¶

The tool to check the availability or syntax of domain, IP or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

Provides our tester worker. This is the description of a single tester worker.

Author:: Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:: https://pyfunceble.github.io/#/special-thanks
Contributors:: https://pyfunceble.github.io/#/contributors
Project link:: https://github.com/funilrys/PyFunceble
Project documentation:: https://pyfunceble.readthedocs.io/en/dev/
Project homepage:: https://pyfunceble.github.io/

License:

Copyright 2017, 2018, 2019, 2020 Nissar Chababy

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

class PyFunceble.cli.processes.workers.tester.TesterWorker(input_queue: Optional[queue.Queue], output_queue: Optional[queue.Queue] = None, global_exit_event: Optional[multiprocessing.context.BaseContext.Event] = None, *, name: Optional[str] = None, daemon: Optional[bool] = None, continuous_integration: Optional[PyFunceble.cli.continuous_integration.base.ContinuousIntegrationBase] = None, configuration: Optional[dict] = None)[source]¶

Bases: PyFunceble.cli.processes.workers.base.WorkerBase

Provides our tester worker. The objective of this worker is to provides a single worker (or process if you prefer) which will be used to handle the tests.

STD_NAME = 'pyfunceble_tester_worker'¶

continue_dataset = None¶

static get_testing_object(subject_type: str, checker_type: str) → PyFunceble.checker.base.CheckerBase[source]¶

Provides the object to use for testing.

Raises:	ValueError – When the given subject type is unkown.

inactive_dataset = None¶

static should_be_ignored(subject: str) → bool[source]¶: Checks if the given subject should be ignored.

target(consumed: dict) → Optional[Tuple[Any, ...]][source]¶: This the target that is run to process something. This method should return a result which will pu send to the output queue.

Module contents¶

The tool to check the availability or syntax of domain, IP or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

Provides all our standalone workers.

Author:: Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:: https://pyfunceble.github.io/#/special-thanks
Contributors:: https://pyfunceble.github.io/#/contributors
Project link:: https://github.com/funilrys/PyFunceble
Project documentation:: https://pyfunceble.readthedocs.io/en/dev/
Project homepage:: https://pyfunceble.github.io/

License:

Copyright 2017, 2018, 2019, 2020 Nissar Chababy

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.