PyFunceble.cli.processes.workers package¶
Submodules¶
PyFunceble.cli.processes.workers.base module¶
The tool to check the availability or syntax of domain, IP or URL.
██████╗ ██╗ ██╗███████╗██╗ ██╗███╗ ██╗ ██████╗███████╗██████╗ ██╗ ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║ ██║████╗ ██║██╔════╝██╔════╝██╔══██╗██║ ██╔════╝
██████╔╝ ╚████╔╝ █████╗ ██║ ██║██╔██╗ ██║██║ █████╗ ██████╔╝██║ █████╗
██╔═══╝ ╚██╔╝ ██╔══╝ ██║ ██║██║╚██╗██║██║ ██╔══╝ ██╔══██╗██║ ██╔══╝
██║ ██║ ██║ ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝ ╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝
Provides the base of all our workers.
- Author:
- Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
- Special thanks:
- https://pyfunceble.github.io/#/special-thanks
- Contributors:
- https://pyfunceble.github.io/#/contributors
- Project link:
- https://github.com/funilrys/PyFunceble
- Project documentation:
- https://pyfunceble.readthedocs.io/en/latest/
- Project homepage:
- https://pyfunceble.github.io/
License:
Copyright 2017, 2018, 2019, 2020, 2022, 2023 Nissar Chababy
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-
class
PyFunceble.cli.processes.workers.base.
WorkerBase
(input_queue: Optional[queue.Queue], output_queue: Optional[queue.Queue] = None, global_exit_event: Optional[multiprocessing.context.BaseContext.Event] = None, *, name: Optional[str] = None, daemon: Optional[bool] = None, continuous_integration: Optional[PyFunceble.cli.continuous_integration.base.ContinuousIntegrationBase] = None, configuration: Optional[dict] = None)[source]¶ Bases:
multiprocessing.context.Process
Provides the base of all our workers.
Parameters: - input_queue – The input queue to read.
- output_queue – The output queue to write.
-
BREAKOFF
= 2.0¶
-
MINING_WAIT_TIME
= 60¶
-
STD_NAME
= 'pyfunceble_base_worker'¶
-
accept_waiting_delay
= None¶
-
add_to_input_queue
(data: Any, *, worker_name: Optional[str] = None, destination_worker: Optional[str] = None) → PyFunceble.cli.processes.workers.base.WorkerBase[source]¶ Adds the given data to the current queue.
Parameters: - data – The data to add into the queue.
- destination_worker – The name of the worker which is supposed to read the message.
-
add_to_output_queue
(data: Any, *, worker_name: Optional[str] = None, destination_worker: Optional[str] = None) → PyFunceble.cli.processes.workers.base.WorkerBase[source]¶ Adds the given data to the output queue queue.
Parameters: data – The data to add into the queue.
-
concurrent_worker_names
= None¶
-
continuous_integration
= None¶
-
db_session
= None¶
-
exception
¶ Provides the exception of the current worker.
-
exit_it
= None¶
-
global_exit_event
= None¶
-
input_queue
= None¶
-
output_queue
= None¶
-
send_stop_message
= None¶
Shares a waiting message to both input and output queues. The idea is to keep our worker awake when necessary. This functionality is really important when running the mining worker because sometime the flow is so long, that we may need some time.
Parameters: - overall – Shares the message multiple time to specific destination in our current stack.
- ignore_current_worker – Doesn’t share the message with ourself.
- apply_breakoff – Activates the breakoff (sleep) before starting to share the message.
PyFunceble.cli.processes.workers.chancy_producer module¶
The tool to check the availability or syntax of domain, IP or URL.
██████╗ ██╗ ██╗███████╗██╗ ██╗███╗ ██╗ ██████╗███████╗██████╗ ██╗ ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║ ██║████╗ ██║██╔════╝██╔════╝██╔══██╗██║ ██╔════╝
██████╔╝ ╚████╔╝ █████╗ ██║ ██║██╔██╗ ██║██║ █████╗ ██████╔╝██║ █████╗
██╔═══╝ ╚██╔╝ ██╔══╝ ██║ ██║██║╚██╗██║██║ ██╔══╝ ██╔══██╗██║ ██╔══╝
██║ ██║ ██║ ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝ ╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝
Provides our producer worker.
- Author:
- Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
- Special thanks:
- https://pyfunceble.github.io/#/special-thanks
- Contributors:
- https://pyfunceble.github.io/#/contributors
- Project link:
- https://github.com/funilrys/PyFunceble
- Project documentation:
- https://pyfunceble.readthedocs.io/en/latest/
- Project homepage:
- https://pyfunceble.github.io/
License:
Copyright 2017, 2018, 2019, 2020, 2022, 2023 Nissar Chababy
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-
class
PyFunceble.cli.processes.workers.chancy_producer.
ChancyProducerWorker
(input_queue: Optional[queue.Queue], output_queue: Optional[queue.Queue] = None, global_exit_event: Optional[multiprocessing.context.BaseContext.Event] = None, *, name: Optional[str] = None, daemon: Optional[bool] = None, continuous_integration: Optional[PyFunceble.cli.continuous_integration.base.ContinuousIntegrationBase] = None, configuration: Optional[dict] = None)[source]¶ Bases:
PyFunceble.cli.processes.workers.producer.ProducerWorker
Provides our chancy producer worker.
Warning
This chancy producer does not provide any guarantee. The flow that keep PyFunceble safe are here unleashed.
USE AT YOUR OWN RISK. GOOD LUCK!
-
STD_NAME
= 'pyfunceble_chancy_producer_worker'¶
-
PyFunceble.cli.processes.workers.chancy_tester module¶
The tool to check the availability or syntax of domain, IP or URL.
██████╗ ██╗ ██╗███████╗██╗ ██╗███╗ ██╗ ██████╗███████╗██████╗ ██╗ ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║ ██║████╗ ██║██╔════╝██╔════╝██╔══██╗██║ ██╔════╝
██████╔╝ ╚████╔╝ █████╗ ██║ ██║██╔██╗ ██║██║ █████╗ ██████╔╝██║ █████╗
██╔═══╝ ╚██╔╝ ██╔══╝ ██║ ██║██║╚██╗██║██║ ██╔══╝ ██╔══██╗██║ ██╔══╝
██║ ██║ ██║ ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝ ╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝
Provides our chancy tester worker. The chancy tester worker is a worker that abstract from the standard tester worker. It just get rid of the walls between some of the component of our data workflow.
- Author:
- Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
- Special thanks:
- https://pyfunceble.github.io/#/special-thanks
- Contributors:
- https://pyfunceble.github.io/#/contributors
- Project link:
- https://github.com/funilrys/PyFunceble
- Project documentation:
- https://pyfunceble.readthedocs.io/en/latest/
- Project homepage:
- https://pyfunceble.github.io/
License:
Copyright 2017, 2018, 2019, 2020, 2022, 2023 Nissar Chababy
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-
class
PyFunceble.cli.processes.workers.chancy_tester.
ChancyTesterWorker
(input_queue: Optional[queue.Queue], output_queue: Optional[queue.Queue] = None, global_exit_event: Optional[multiprocessing.context.BaseContext.Event] = None, *, name: Optional[str] = None, daemon: Optional[bool] = None, continuous_integration: Optional[PyFunceble.cli.continuous_integration.base.ContinuousIntegrationBase] = None, configuration: Optional[dict] = None)[source]¶ Bases:
PyFunceble.cli.processes.workers.tester.TesterWorker
Provides our chancy tester worker. The chancy worker breaks the walls between some of the core component of our data workflow.
Warning
This chancy tester does not provide any guarantee. The flow that keep PyFunceble safe are here unleashed.
USE AT YOUR OWN RISK. GOOD LUCK!
-
STD_NAME
= 'pyfunceble_chancy_tester_worker'¶
-
PyFunceble.cli.processes.workers.dir_files_sorter module¶
The tool to check the availability or syntax of domain, IP or URL.
██████╗ ██╗ ██╗███████╗██╗ ██╗███╗ ██╗ ██████╗███████╗██████╗ ██╗ ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║ ██║████╗ ██║██╔════╝██╔════╝██╔══██╗██║ ██╔════╝
██████╔╝ ╚████╔╝ █████╗ ██║ ██║██╔██╗ ██║██║ █████╗ ██████╔╝██║ █████╗
██╔═══╝ ╚██╔╝ ██╔══╝ ██║ ██║██║╚██╗██║██║ ██╔══╝ ██╔══██╗██║ ██╔══╝
██║ ██║ ██║ ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝ ╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝
Provides our directory files sorter worker. This is the description of a single directory file sorter worker.
- Author:
- Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
- Special thanks:
- https://pyfunceble.github.io/#/special-thanks
- Contributors:
- https://pyfunceble.github.io/#/contributors
- Project link:
- https://github.com/funilrys/PyFunceble
- Project documentation:
- https://pyfunceble.readthedocs.io/en/latest/
- Project homepage:
- https://pyfunceble.github.io/
License:
Copyright 2017, 2018, 2019, 2020, 2022, 2023 Nissar Chababy
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-
class
PyFunceble.cli.processes.workers.dir_files_sorter.
DireFileSorterWorker
(input_queue: Optional[queue.Queue], output_queue: Optional[queue.Queue] = None, global_exit_event: Optional[multiprocessing.context.BaseContext.Event] = None, *, name: Optional[str] = None, daemon: Optional[bool] = None, continuous_integration: Optional[PyFunceble.cli.continuous_integration.base.ContinuousIntegrationBase] = None, configuration: Optional[dict] = None)[source]¶ Bases:
PyFunceble.cli.processes.workers.file_sorter_base.FileSorterWorkerBase
Provides our directory files sorter worker. The objective of this worker is to provides a single worker (or process if you prefer) which will be used to handle the sorting of the content of a submitted directory.
Expected (input) message:
{ "directory": str, "remove_duplicates": bool, "write_header": bool }
Expected (output) message:
None
-
STD_NAME
= 'pyfunceble_dir_files_sorter_worker'¶
-
PyFunceble.cli.processes.workers.file_sorter module¶
The tool to check the availability or syntax of domain, IP or URL.
██████╗ ██╗ ██╗███████╗██╗ ██╗███╗ ██╗ ██████╗███████╗██████╗ ██╗ ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║ ██║████╗ ██║██╔════╝██╔════╝██╔══██╗██║ ██╔════╝
██████╔╝ ╚████╔╝ █████╗ ██║ ██║██╔██╗ ██║██║ █████╗ ██████╔╝██║ █████╗
██╔═══╝ ╚██╔╝ ██╔══╝ ██║ ██║██║╚██╗██║██║ ██╔══╝ ██╔══██╗██║ ██╔══╝
██║ ██║ ██║ ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝ ╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝
Provides our file sorter worker. This is the description of a file sorter worker.
- Author:
- Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
- Special thanks:
- https://pyfunceble.github.io/#/special-thanks
- Contributors:
- https://pyfunceble.github.io/#/contributors
- Project link:
- https://github.com/funilrys/PyFunceble
- Project documentation:
- https://pyfunceble.readthedocs.io/en/latest/
- Project homepage:
- https://pyfunceble.github.io/
License:
Copyright 2017, 2018, 2019, 2020, 2022, 2023 Nissar Chababy
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-
class
PyFunceble.cli.processes.workers.file_sorter.
FileSorterWorker
(input_queue: Optional[queue.Queue], output_queue: Optional[queue.Queue] = None, global_exit_event: Optional[multiprocessing.context.BaseContext.Event] = None, *, name: Optional[str] = None, daemon: Optional[bool] = None, continuous_integration: Optional[PyFunceble.cli.continuous_integration.base.ContinuousIntegrationBase] = None, configuration: Optional[dict] = None)[source]¶ Bases:
PyFunceble.cli.processes.workers.file_sorter_base.FileSorterWorkerBase
Provides our file sorter worker. The objective of this worker is to provides a single worker (or process if you prefer) which will be used to handle the sorting of the content of a given file.
Expected (input) message:
{ "file": str, "remove_duplicates": bool, "write_header": bool }
Expected (output) message:
None
-
STD_NAME
= 'pyfunceble_file_sorter_worker'¶
-
PyFunceble.cli.processes.workers.file_sorter_base module¶
The tool to check the availability or syntax of domain, IP or URL.
██████╗ ██╗ ██╗███████╗██╗ ██╗███╗ ██╗ ██████╗███████╗██████╗ ██╗ ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║ ██║████╗ ██║██╔════╝██╔════╝██╔══██╗██║ ██╔════╝
██████╔╝ ╚████╔╝ █████╗ ██║ ██║██╔██╗ ██║██║ █████╗ ██████╔╝██║ █████╗
██╔═══╝ ╚██╔╝ ██╔══╝ ██║ ██║██║╚██╗██║██║ ██╔══╝ ██╔══██╗██║ ██╔══╝
██║ ██║ ██║ ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝ ╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝
Provides our file sorter worker base. This is the base of all our file sorter.
- Author:
- Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
- Special thanks:
- https://pyfunceble.github.io/#/special-thanks
- Contributors:
- https://pyfunceble.github.io/#/contributors
- Project link:
- https://github.com/funilrys/PyFunceble
- Project documentation:
- https://pyfunceble.readthedocs.io/en/latest/
- Project homepage:
- https://pyfunceble.github.io/
License:
Copyright 2017, 2018, 2019, 2020, 2022, 2023 Nissar Chababy
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-
class
PyFunceble.cli.processes.workers.file_sorter_base.
FileSorterWorkerBase
(input_queue: Optional[queue.Queue], output_queue: Optional[queue.Queue] = None, global_exit_event: Optional[multiprocessing.context.BaseContext.Event] = None, *, name: Optional[str] = None, daemon: Optional[bool] = None, continuous_integration: Optional[PyFunceble.cli.continuous_integration.base.ContinuousIntegrationBase] = None, configuration: Optional[dict] = None)[source]¶ Bases:
PyFunceble.cli.processes.workers.base.WorkerBase
Provides our the base of all our file sorters.
-
FILE_BUFFER_SIZE
= 65536¶
-
MAX_LINES
= 32000¶
-
classmethod
process_file_sorting
(file: str, remove_duplicates: bool = True, write_header: bool = True, sorting_key: Any = None) → None[source]¶ Process the sorting of the given file.
The idea is to split the file piece by piece and at the end join all sorted files. For that job, we create a temporary directory which will store the temporary files.
Parameters: - file – The file to sort.
- remove_duplicates – Activates the deletion of duplicates.
- write_header –
Activates the writing of the PyFunceble related header.
Warning
When this is set to
True
, we assume that the header itself was already given. Meaning that the first 2 commented lines will be excluded from the sorting and regenerated. - sorting_key –
The sorting key to apply while sorting.
This is the lambda/function that goes into the
key
argument of thesorted
function.
-
PyFunceble.cli.processes.workers.migrator module¶
The tool to check the availability or syntax of domain, IP or URL.
██████╗ ██╗ ██╗███████╗██╗ ██╗███╗ ██╗ ██████╗███████╗██████╗ ██╗ ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║ ██║████╗ ██║██╔════╝██╔════╝██╔══██╗██║ ██╔════╝
██████╔╝ ╚████╔╝ █████╗ ██║ ██║██╔██╗ ██║██║ █████╗ ██████╔╝██║ █████╗
██╔═══╝ ╚██╔╝ ██╔══╝ ██║ ██║██║╚██╗██║██║ ██╔══╝ ██╔══██╗██║ ██╔══╝
██║ ██║ ██║ ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝ ╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝
Provides our migrator (base) worker.
- Author:
- Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
- Special thanks:
- https://pyfunceble.github.io/#/special-thanks
- Contributors:
- https://pyfunceble.github.io/#/contributors
- Project link:
- https://github.com/funilrys/PyFunceble
- Project documentation:
- https://pyfunceble.readthedocs.io/en/latest/
- Project homepage:
- https://pyfunceble.github.io/
License:
Copyright 2017, 2018, 2019, 2020, 2022, 2023 Nissar Chababy
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-
class
PyFunceble.cli.processes.workers.migrator.
MigratorWorker
(input_queue: Optional[queue.Queue], output_queue: Optional[queue.Queue] = None, global_exit_event: Optional[multiprocessing.context.BaseContext.Event] = None, *, name: Optional[str] = None, daemon: Optional[bool] = None, continuous_integration: Optional[PyFunceble.cli.continuous_integration.base.ContinuousIntegrationBase] = None, configuration: Optional[dict] = None)[source]¶ Bases:
PyFunceble.cli.processes.workers.base.WorkerBase
Provides our migrator (base) worker. The objective of this worker is to provides a single worker (or process if you prefer) which will be used to handle the mining of dataset to test.
-
STD_NAME
= 'pyfunceble_migrator_worker'¶
-
PyFunceble.cli.processes.workers.miner module¶
The tool to check the availability or syntax of domain, IP or URL.
██████╗ ██╗ ██╗███████╗██╗ ██╗███╗ ██╗ ██████╗███████╗██████╗ ██╗ ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║ ██║████╗ ██║██╔════╝██╔════╝██╔══██╗██║ ██╔════╝
██████╔╝ ╚████╔╝ █████╗ ██║ ██║██╔██╗ ██║██║ █████╗ ██████╔╝██║ █████╗
██╔═══╝ ╚██╔╝ ██╔══╝ ██║ ██║██║╚██╗██║██║ ██╔══╝ ██╔══██╗██║ ██╔══╝
██║ ██║ ██║ ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝ ╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝
Provides our miner worker. This is the description of a single miner worker.
- Author:
- Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
- Special thanks:
- https://pyfunceble.github.io/#/special-thanks
- Contributors:
- https://pyfunceble.github.io/#/contributors
- Project link:
- https://github.com/funilrys/PyFunceble
- Project documentation:
- https://pyfunceble.readthedocs.io/en/latest/
- Project homepage:
- https://pyfunceble.github.io/
License:
Copyright 2017, 2018, 2019, 2020, 2022, 2023 Nissar Chababy
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-
class
PyFunceble.cli.processes.workers.miner.
MinerWorker
(input_queue: Optional[queue.Queue], output_queue: Optional[queue.Queue] = None, global_exit_event: Optional[multiprocessing.context.BaseContext.Event] = None, *, name: Optional[str] = None, daemon: Optional[bool] = None, continuous_integration: Optional[PyFunceble.cli.continuous_integration.base.ContinuousIntegrationBase] = None, configuration: Optional[dict] = None)[source]¶ Bases:
PyFunceble.cli.processes.workers.base.WorkerBase
Provides our miner worker. The objective of this worker is to provides a single worker (or process if you prefer) which will be used to handle the mining of dataset to test.
-
INACTIVE_STATUSES
= ('INACTIVE', 'INVALID')¶
-
STD_NAME
= 'pyfunceble_miner_worker'¶
-
static
mine_from
(subject: str) → Optional[List[str]][source]¶ Given the subject to work from, try to get the related subjects.
Parameters: subject – The URL to start from.
-
target
(consumed: Tuple[dict, PyFunceble.checker.status_base.CheckerStatusBase]) → None[source]¶ This the target that is run to process something. This method should return a result which will pu send to the output queue.
-
url2netloc
= None¶
-
PyFunceble.cli.processes.workers.producer module¶
The tool to check the availability or syntax of domain, IP or URL.
██████╗ ██╗ ██╗███████╗██╗ ██╗███╗ ██╗ ██████╗███████╗██████╗ ██╗ ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║ ██║████╗ ██║██╔════╝██╔════╝██╔══██╗██║ ██╔════╝
██████╔╝ ╚████╔╝ █████╗ ██║ ██║██╔██╗ ██║██║ █████╗ ██████╔╝██║ █████╗
██╔═══╝ ╚██╔╝ ██╔══╝ ██║ ██║██║╚██╗██║██║ ██╔══╝ ██╔══██╗██║ ██╔══╝
██║ ██║ ██║ ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝ ╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝
Provides our producer worker. This is the description of a single producer worker.
- Author:
- Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
- Special thanks:
- https://pyfunceble.github.io/#/special-thanks
- Contributors:
- https://pyfunceble.github.io/#/contributors
- Project link:
- https://github.com/funilrys/PyFunceble
- Project documentation:
- https://pyfunceble.readthedocs.io/en/latest/
- Project homepage:
- https://pyfunceble.github.io/
License:
Copyright 2017, 2018, 2019, 2020, 2022, 2023 Nissar Chababy
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-
class
PyFunceble.cli.processes.workers.producer.
ProducerWorker
(input_queue: Optional[queue.Queue], output_queue: Optional[queue.Queue] = None, global_exit_event: Optional[multiprocessing.context.BaseContext.Event] = None, *, name: Optional[str] = None, daemon: Optional[bool] = None, continuous_integration: Optional[PyFunceble.cli.continuous_integration.base.ContinuousIntegrationBase] = None, configuration: Optional[dict] = None)[source]¶ Bases:
PyFunceble.cli.processes.workers.base.WorkerBase
Provides our producer worker. The objective of this worker is to provides a single worker (or process if you prefer) which will be used to handle the production of output to stdout or files.
-
INACTIVE_STATUSES
= ('INACTIVE', 'INVALID')¶
-
STD_NAME
= 'pyfunceble_producer_worker'¶
-
collection_query_tool
= None¶
-
continue_dataset
= None¶
-
counter
= None¶
-
file_printer
= None¶
-
header_already_printed
= None¶
-
inactive_dataset
= None¶
-
registrar_counter
= None¶
-
run_continue_backup
(test_dataset: dict, test_result: PyFunceble.checker.status_base.CheckerStatusBase) → None[source]¶ Runs the backup or update of the auto-continue dataset storage.
-
run_counter
(test_dataset: dict, test_result: PyFunceble.checker.status_base.CheckerStatusBase) → None[source]¶ Runs the counter of the current file.
-
run_ignored_file_printer
(test_dataset: dict, test_result: str) → None[source]¶ Runs the analytic behind the file printer.
Warning
Thie method assume that the givne dataset is ignored from the normal file printer.
-
run_inactive_backup
(test_dataset: dict, test_result: PyFunceble.checker.status_base.CheckerStatusBase) → None[source]¶ Runs the backup or update of the Inactive dataset storage.
The idea is that if the status is OK (active), we just remove it from the dataset storage. Otherwise, we just keep it in there :-)
-
run_status_file_printer
(test_dataset: dict, test_result: PyFunceble.checker.status_base.CheckerStatusBase) → None[source]¶ Runs the status file printer.
-
run_stdout_printer
(test_result: PyFunceble.checker.status_base.CheckerStatusBase) → None[source]¶ Runs the stdout printer (if necessary).
Parameters: test_result – The rest result dataset.
-
run_whois_backup
(test_result: PyFunceble.checker.status_base.CheckerStatusBase) → None[source]¶ Runs the backup or update of the WHOIS record in our dataset storage.
Parameters: test_result – The test result.
-
should_we_block_status_file_printer
(test_dataset: dict, test_result: PyFunceble.checker.status_base.CheckerStatusBase) → bool[source]¶ Checks if we should block the file printer.
The reason behindn this is that we don’t want to generate an output when a subject was already into the inactive database.
-
static
should_we_ignore
(test_result: PyFunceble.checker.status_base.CheckerStatusBase) → bool[source]¶ Checks if we should ignore the given datasets.
Parameters: test_result – The test result to check.
-
static
should_we_print_status_to_stdout
(status: str) → bool[source]¶ Checks if we are allows to print the given status (to stdout).
Parameters: status – The status to check.
-
status_file_generator
= None¶
-
stdout_printer
= None¶
-
target
(consumed: Any) → Optional[Tuple[Any, ...]][source]¶ This the target that is run to process something. This method should return a result which will pu send to the output queue.
-
whois_dataset
= None¶
-
PyFunceble.cli.processes.workers.tester module¶
The tool to check the availability or syntax of domain, IP or URL.
██████╗ ██╗ ██╗███████╗██╗ ██╗███╗ ██╗ ██████╗███████╗██████╗ ██╗ ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║ ██║████╗ ██║██╔════╝██╔════╝██╔══██╗██║ ██╔════╝
██████╔╝ ╚████╔╝ █████╗ ██║ ██║██╔██╗ ██║██║ █████╗ ██████╔╝██║ █████╗
██╔═══╝ ╚██╔╝ ██╔══╝ ██║ ██║██║╚██╗██║██║ ██╔══╝ ██╔══██╗██║ ██╔══╝
██║ ██║ ██║ ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝ ╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝
Provides our tester worker. This is the description of a single tester worker.
- Author:
- Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
- Special thanks:
- https://pyfunceble.github.io/#/special-thanks
- Contributors:
- https://pyfunceble.github.io/#/contributors
- Project link:
- https://github.com/funilrys/PyFunceble
- Project documentation:
- https://pyfunceble.readthedocs.io/en/latest/
- Project homepage:
- https://pyfunceble.github.io/
License:
Copyright 2017, 2018, 2019, 2020, 2022, 2023 Nissar Chababy
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-
class
PyFunceble.cli.processes.workers.tester.
TesterWorker
(input_queue: Optional[queue.Queue], output_queue: Optional[queue.Queue] = None, global_exit_event: Optional[multiprocessing.context.BaseContext.Event] = None, *, name: Optional[str] = None, daemon: Optional[bool] = None, continuous_integration: Optional[PyFunceble.cli.continuous_integration.base.ContinuousIntegrationBase] = None, configuration: Optional[dict] = None)[source]¶ Bases:
PyFunceble.cli.processes.workers.base.WorkerBase
Provides our tester worker. The objective of this worker is to provides a single worker (or process if you prefer) which will be used to handle the tests.
-
STD_NAME
= 'pyfunceble_tester_worker'¶
-
continue_dataset
= None¶
-
inactive_dataset
= None¶
-
initiated_testing_objects
= {}¶
-
known_testing_objects
= {}¶
-
static
should_be_ignored
(subject: str) → bool[source]¶ Checks if the given subject should be ignored.
-
target
(consumed: dict) → Optional[Tuple[Any, ...]][source]¶ This the target that is run to process something. This method should return a result which will pu send to the output queue.
-
testing_object
= None¶
-
Module contents¶
The tool to check the availability or syntax of domain, IP or URL.
██████╗ ██╗ ██╗███████╗██╗ ██╗███╗ ██╗ ██████╗███████╗██████╗ ██╗ ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║ ██║████╗ ██║██╔════╝██╔════╝██╔══██╗██║ ██╔════╝
██████╔╝ ╚████╔╝ █████╗ ██║ ██║██╔██╗ ██║██║ █████╗ ██████╔╝██║ █████╗
██╔═══╝ ╚██╔╝ ██╔══╝ ██║ ██║██║╚██╗██║██║ ██╔══╝ ██╔══██╗██║ ██╔══╝
██║ ██║ ██║ ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝ ╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝
Provides all our standalone workers.
- Author:
- Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
- Special thanks:
- https://pyfunceble.github.io/#/special-thanks
- Contributors:
- https://pyfunceble.github.io/#/contributors
- Project link:
- https://github.com/funilrys/PyFunceble
- Project documentation:
- https://pyfunceble.readthedocs.io/en/latest/
- Project homepage:
- https://pyfunceble.github.io/
License:
Copyright 2017, 2018, 2019, 2020, 2022, 2023 Nissar Chababy
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.