Category: Coding

Multiprocessing using Python 3.7

Multiprocessing using Python 3.7

Hey Guys,
I’d like to tell you about Multiprocessing using Python 3.7 which is slightly different that Multithreading with Python 2.7. I already wrote something about Multithreading with Python 2.7 which you can read here.

Basics

By definition a process is a collection of one or more threads that shares memory, code segments and rights but do not share with another processes.

Accordingly to prior paragraph the default case of using multiprocessing is when your program can be divided into several tasks running concurrently and independent from each other. In addition, multithreading is normally used for shared data structure that is written and read by various threads within a process.

I prepared two examples to you how to treat a Multiprocessing pool synchronous or asynchronous.

Examples

The following code block shows the basic code I used in both examples. I only changed the way how to wait until all child processes are done.

#!/usr/bin/env python3
## Importing all needed modules
import multiprocessing
import random
import time
import timeit

print("### Starting Multiprocessing")

## Starting timer which keep track of the runtime
start_time = timeit.default_timer()

## Define the function which will be executed within the pool
def asyncProcess(processRuntime, processID):
    """
    Sleeps the amount of time as seconds from variable processRuntime
    returns: <STRING>
    """
    time.sleep(processRuntime)
    return("%i: This Process ran %i seconds" % (processID, processRuntime))


if __name__ == '__main__':
    ## Define Pool Size of maximal concurrent processes
    pool_size = 10

    ## Define an empty list where we store child processes
    processes = []

    ## Define an empty list to store finished processes
    finished_processes = []

    ## Define an empty pool whith maximal concurrent processes
    pool = multiprocessing.Pool(processes=pool_size)

    ## Firing in total of 10 processes
    for i in range(0, pool_size):
        ## Append the process variable with a process object
        processes.append(pool.apply_async(asyncProcess, args=(random.randint(0, 10), i,)))
    
    ## Closes the pool to stop accepting new processes
    pool.close()

Synchronous Pool

Sometimes you want to execute some tasks but want to wait until all child process are finished. To achieve this you need to execute pool.join() which simply waits until all child processes finished their job. This is similar to Multithreading in Python 2.7.

    pool.join()
    ## Iterate through the processes variable
    for process in processes:
        ## Print the process returned value
        print(process.get())
            
    print("Parent: this Process ran %s seconds"
          % str(timeit.default_timer() - start_time))

Demo

Asynchronous Pool

You are also able to e.g. print the result if a child process is done. Alternatively, you could do something further with the data until everything else is finished.

    ## Iterate through processes as long as a process is running
    while True:
        ## Iterate through the processes variable
        for process in processes:
            ## Check if process is done and not in finished_processes
            if(process.ready() and process not in finished_processes):
                ## Print the returned value
                print(process.get())
                ## Append the finished process to finished_processes
                finished_processes.append(process)
        ## Break while loop when finished_processes length equal processes length
        if len(finished_processes) == len(processes):
            break
    print("Parent: this Process ran %s seconds" % str(timeit.default_timer() - start_time))

Demo

Conclusion

I hope you enjoyed this small insight into Pythons Multiprocessing. If you have any questions or improvements just leave a comment below this post.

If you want to read more about Multiprocessing and Processes in general, take a look at these links:

Title Image via LINK

Catching a KeyboardInterrupt Signal

Catching a KeyboardInterrupt Signal

Hey everyone,

as I thought about writing the next post I came across the idea to explain what KeyboardInterrupt actually is and also how to catch it in Python 3.6.

KeyboardInterrupt is a User Signal which is executed by User. As an example could be pressing CTRL+C. In detail, the Signal is called SIGINT (Signal Interrupt). It could be useful catching a KeyboardInterrupt Signal to e.g. cleaning up temporary files which have been created during script execution.

Requirements

At first, we need to create a python script. Let’s name it ‘keyboardinterrupt.py’.

[email protected]: touch keyboardinterrupt.py

Catching SIGINT

Catching the SIGINT Signal may be useful for scripts or application which are multithreaded or when it’s necessary to be caught during the whole script execution.

Prerequisites

To catch the SIGINT it is required to import the signal library.

Importing signal
#!/usr/bin/env python3
import signal
Handler for the caught KeyboardInterrupt SIGINT

Now we need to add a function which is executed as soon as the SIGINT has been caught.

Signal Handler Function
def keyboardInterruptHandler(signal, frame):
    print("KeyboardInterrupt (ID: {}) has been caught. Cleaning up...".format(signal))
    exit(0)

As you can see, the function takes two positional arguments. The signal (integer) and the frame (object). The signal variable is an integer which represents the number of the caught signal. For SIGINT it’s the two (2). The frame variable represents the current Python stack frame which can be useful for debugging purposes.

Registering the signal action

To execute a function before quitting the program you need to register an action.

Action registering
signal.signal(signal.SIGINT, keyboardInterruptHandler)
Executing a long-running task

Now we’re executing a long-running task which will be interrupted by SIGINT.

Long-running task
while True:
    pass

Please note that the script will never end on it’s own.

Putting it all together
Whole script
#!/usr/bin/env python3
import signal

def keyboardInterruptHandler(signal, frame):
    print("KeyboardInterrupt (ID: {}) has been caught. Cleaning up...".format(signal))
    exit(0)

signal.signal(signal.SIGINT, keyboardInterruptHandler)

while True:
    pass

When you run the script from above you will notice that nothing is happening. Just press CTRL+C and you will see the print output from the highlighted line.

Running the script

[email protected]: python3 keyboardinterrupt.py
<PRESS CTRL+C NOW!!!>
^CKeyboardInterrupt (ID: 2) has been caught. Cleaning up…

Catching KeyboardInterrupt exception

The second method reacting to is catching the KeyboardInterrupt exception. It can be useful in a script which is single threaded and also in long-running task situations which can be skipped but not canceling the whole script.

Prerequisites

We use the time library to simulate a running task. Therefore we’re importing it into a new file.

Importing time library
#!/usr/bin/env python3
import time
Adding a long-running task in a “try” block

We’re catching the KeyboardInterrupt exception with a “try-except” block.

try-except block
try:
    while True:
        print("Heavy task!")
        time.sleep(2)
except KeyboardInterrupt:
    print("KeyboardInterrupt has been caught.")
Putting it all together
Catching the exception
#!/usr/bin/env python3
import time

print("Calculating a lot... Can be canceled with CTRL+C")
try:
    while True:
        print("Heavy task!")
        time.sleep(2)
except KeyboardInterrupt:
    print("KeyboardInterrupt has been caught.")

print("Continuing script execution")

As you can see i added some print information to add a hint to a user that the current task in the script can be skipped.

Running the script

[email protected]: python3 keyboardinterrupt.py
Calculating a lot… Can be canceled with CTRL+C
Heavy task!
Heavy task!
Heavy task!
Heavy task!
Heavy task!
^CKeyboardInterrupt has been caught.
Continuing script execution

Conclusion

It’s very easy to implement signal catching in python 3.6. You can easily catch any kind of signal and react to it in a matching manner. As an example taken from dd (this is a unix program to convert and copy files) it is required to send USRSIG1 to the running process to show the current state of the progress.

Do you have any interesting ideas what to do with caught signals?

Further Ressources

Hacking Rocketbeans lamp with ESP32 to display current show type

Hacking Rocketbeans lamp with ESP32 to display current show type

Hey guys,

today I’d like to show you how I modded an ordinary lamp into a smart (IoT) one.

Foreword

I’m a huge Rocketbeans TV Fan. Rocketbeans is a big 24/7 Stream Show focused on gaming. Germans may know the hosts from “Giga” or “Game One”.

As they started selling a USB Lamp with their logo and some RGB-LED I thought about making it smart to display the current type of show running. At the moment there are three types of shows available: “Live”(=Red), “Premiere”(=Blue), “Playback”(=White). The unmodified lamp is able to switch colors on a button press.

via Rocketbeans Forum

Requirements

The Lamp

I bought one of them. As soon as it arrived I started to disassemble the lamp to check the circuit board. To my surprise, it’s straight forward.

The (smart) Microcontroller

Leroy told me about the ESP32 which is a very small Microcontroller with built-in Wifi and Bluetooth. This one seems to be a perfect match for the smart part of the lamp.

Loading

 

You can buy one via eBay

The Schedule of Rocketbeans TV

After viewing and manually scraping their Website I was able to find a Link which returns the current and four further shows as JSON object. (The following image only shows the current show also referred as Array[0]).

Now we gathered all requirements and start tinkering.

Reverse Engineering the Lamp Circuit

We started to measure some circuits and figured out how the embedded controller works. The controller has eight pins: GND, 5v, a Touch Button on the Case and three colors. The remaining ones may be used for infrared.

Next we removed the embedded microcontroller to attach the ESP32 controller.

Writing the Microcontroller Code

Since I never wrote any kind of ESP32 Code I decided to invest some time to read Tutorials/Blogs about “ESP32 Wifi”, “Arduino JSON” and “Arduino HTTP Client”.

After several compile errors and reading through manuals on how to use the above libraries I wrote the following code.

ESP32 Code (Version 1)
#include <WiFi.h>
#include <HTTPClient.h>
#include <ArduinoJson.h>

#define __product__    "Beanduino: Dynamic Color Lamp"
#define __model__      "DyCoLa"
#define __version__    "1000DEV"
#define __author__     "Marvyn Zalewski <[email protected]>"

#define __copyright__ "(c) 2015 The Senso Team"
#define __license__   "All rights reserved."

const char* ssid = "WIFI";
const char* password =  "WIFIPASSWORD";
const char* rbTvUri = "https://www.rocketbeans.tv/?next5Shows=true";
int status = WL_IDLE_STATUS;


void setup() {
  Serial.begin(115200);
  Serial.println(__product__);
  Serial.print(__model__);
  Serial.println(__version__);
  Serial.println(__author__);
  Serial.println((String)__copyright__ + ", " + (String)__license__ + "\n");
  while ( status != WL_CONNECTED) {
    Serial.println("Beanduino - Attempting to connect to Wifi network, SSID: " + String(ssid));
    status = WiFi.begin(ssid, password);
    delay(10000);
  }
  Serial.println("Beanduino - Connected to network");

}

String stateToColor(String state){
  const size_t showStateJsonBufferSize = JSON_OBJECT_SIZE(3);
  DynamicJsonBuffer showStateJsonBuffer(showStateJsonBufferSize);
  JsonObject& showStates = showStateJsonBuffer.createObject();
  showStates["live"] = "red";
  showStates["new"] = "blue";
  showStates["playback"] = "white";
  return showStates[state];
}

void setLampState(String state){
  String showState = state;
  String showColor = stateToColor(state);
  Serial.println(__product__ ";" + showState + ";" + showColor);
}

void loop() {
  if ((WiFi.status() == WL_CONNECTED)) {
      HTTPClient http;
      http.begin(rbTvUri);
      int httpCode = http.GET();
      if (httpCode > 0) {
          const size_t bufferSize = JSON_ARRAY_SIZE(5) + 5*JSON_OBJECT_SIZE(14) + 2970;
          DynamicJsonBuffer jsonBuffer(bufferSize);
          String payload = http.getString();
          JsonArray& root = jsonBuffer.parseArray(payload);
          if (!root.success()){
              Serial.println("PARSING FAILED");
              Serial.println(payload);
          }
          JsonArray& root_ = root;
          JsonObject& root_0 = root_[0];
          int isLive = root_0["isLive"];
          int isNew = root_0["isNew"];
          if (isLive == 1) {
              setLampState("live");
            } else {
            if (isNew == 1) {
                setLampState("new");
            } else {
                setLampState("playback");
             }    
          }
      }
      else {
        Serial.println("Error on HTTP request");
      }
      http.end();
    }
    delay(10000);
}

Running the code

Beanduino: Dynamic Color Lamp
DyCoLa1000DEV
Marvyn Zalewski <[email protected]>
(c) 2015 The Senso Team, All rights reserved.

Beanduino – Attempting to connect to Wifi network, SSID: WIFI
Beanduino – Attempting to connect to Wifi network, SSID: WIFI
Beanduino – Connected to network
Beanduino: Dynamic Color Lamp;live;red
Beanduino: Dynamic Color Lamp;live;red

Putting it all together

Now we know everything to set up the smart lamp. At first, we’re soldering all needed wires and stick them into the battery compartment.

As you can see there are six wires coming from the Lamp Circuit. (VCC which is 5V, GND, directly from the Button and all three colors.)

Due to space issues, we soldered the wires directly onto the ESP32.

Wire Map:

  • ESP32 VIN -> Lamp Circuit VCC
  • ESP32 GND -> Lamp Circuit GND
  • ESP32 D14 -> Lamp Circuit LED Green (the left resistor R5)
  • ESP32 D22 -> Lamp Case Button (without any function at the moment)
  • ESP32 D26 -> Lamp Circuit LED Blue (the middle resistor R6)
  • ESP32 D33 -> Lamp Circuit LED Red (the right resistor R7)

 

After a nearly whole code refactoring, I finished it and moved it to Github.

Beanlamp Code
#include <Arduino.h>
#include <WiFi.h>
#include <HTTPClient.h>
#include <ArduinoJson.h>

/*
* Coniguration
** CONNECTION_INDICATOR - blinking led during startup when defined
*/
#define CONNECTION_INDICATOR
#define LED_RED GPIO_NUM_14
#define LED_GREEN GPIO_NUM_27
#define LED_BLUE GPIO_NUM_26

#define WIFI_SSID "WLAN"
#define WIFI_PASSWORD "WLAN_PASSWORD"
#define ROCKETBEANS_API "https://www.rocketbeans.tv/?next5Shows=true"

/*
* Information
*/
#define __product__ "Beanlight"
#define __version__ "1000DEV"
#define __author__ "Marvyn Zalewski <[email protected]>"
#define __copyright__ "(c) 2018 KeyboardInterrupt.org"

/*
* Global Variables
*/
int status = WL_IDLE_STATUS;

/*
* Functions
*/
void setLampToRed()
{
    digitalWrite(LED_RED, HIGH);
    digitalWrite(LED_GREEN, LOW);
    digitalWrite(LED_BLUE, LOW);
}

void setLampToBlue()
{
    digitalWrite(LED_BLUE, HIGH);
    digitalWrite(LED_GREEN, LOW);
    digitalWrite(LED_RED, LOW);
}

void setLampToGreen()
{
    digitalWrite(LED_GREEN, HIGH);
    digitalWrite(LED_BLUE, LOW);
    digitalWrite(LED_RED, LOW);
}

void setLampToWhite()
{
    digitalWrite(LED_GREEN, HIGH);
    digitalWrite(LED_BLUE, HIGH);
    digitalWrite(LED_RED, HIGH);
}

void setLampToBlack()
{
    digitalWrite(LED_GREEN, LOW);
    digitalWrite(LED_BLUE, LOW);
    digitalWrite(LED_RED, LOW);
}

/*
* Init
*/
void setup()
{
    Serial.begin(115200);
    Serial.println(__product__);
    Serial.println(__version__);
    Serial.println(__author__);
    Serial.println((String)__copyright__ + "\n");
    pinMode(LED_RED, OUTPUT);
    pinMode(LED_BLUE, OUTPUT);
    pinMode(LED_GREEN, OUTPUT);
    while (status != WL_CONNECTED)
    {
        Serial.println((String)__product__ + " attempting to connect to Wifi network, WIFI_SSID: " + (String)WIFI_SSID);
        status = WiFi.begin(WIFI_SSID, WIFI_PASSWORD);
        #ifdef CONNECTION_INDICATOR
        setLampToRed();
        delay(2000);
        setLampToBlue();
        delay(2000);
        setLampToGreen();
        delay(2000);
        #else
        delay(6000);
        #endif
    };
    #ifdef CONNECTION_INDICATOR
    for (int i = 0; i <= 3; i++)
    {
        setLampToGreen();
        delay(250);
        setLampToBlack();
        delay(250);
    };
    #endif
    Serial.println((String)__product__ + " connected to network");
}

void loop()
{
    if ((WiFi.status() == WL_CONNECTED))
    {
        HTTPClient http;
        http.begin(ROCKETBEANS_API);
        int httpCode = http.GET();
        if (httpCode > 0)
        {
            const size_t bufferSize = JSON_ARRAY_SIZE(5) + 5 * JSON_OBJECT_SIZE(14) + 2970;
            DynamicJsonBuffer jsonBuffer(bufferSize);
            String payload = http.getString();
            JsonArray &root = jsonBuffer.parseArray(payload);
            if (!root.success())
            {
                Serial.println((String)__product__ + " parsing failed.");
                Serial.println(payload);
            }
            /* When current show is live */
            if (root[0]["isLive"] == 1)
                setLampToRed();
            /* When current show is new */
            else if (root[0]["isNew"] == 1)
                setLampToBlue();
            /* When current show is playback */
            else
                setLampToWhite();
        }
        else
        {
            Serial.println((String)__product__ + " error on HTTP request");
        }
        http.end();
    }
    delay(60000);
}

Result

After working nearly ten hours on the project I would say it was a lot of fun.

If you have any questions or improvements, let me know in the comments. 🙂

Rendering HTML with Jinja2 in Python 3.6

Rendering HTML with Jinja2 in Python 3.6

Hey folks,

today I’d like to show how to render an HTML (or any random non-binary) File with Jinja2 and Python 3.6.

A possible usage for rendering an HTML file is the avoidance of writing duplicate code. I prepared a small HTML file which only consists of a headline and the variable.

Source Template
<h1>{{ greeting }}</h1>

As you can see between the h1 tags the text is padded in double curly braces which is the syntax to render variables with Jinja2.

 

Clearing all conditions

At first, we’re going to install Jinja2 which is needed for Rendering.

Installing Jinja2
[email protected]: pip3 install jinja2

 

Now we’re heading to a new file which will be named renderer.py and start with our imports.

Importing needed modules
#!/usr/bin/env python3
import os
import jinja2
import sys

 

After importing all needed modules we’re going to set up some variables.

setting up Variables
template_filename = "template.html.j2"
rendered_filename = "rendered.html"
render_vars = {
    "greeting": "Oh hi!"
}

script_path = os.path.dirname(os.path.abspath(__file__))
template_file_path = os.path.join(current_path, template_filename)
rendered_file_path = os.path.join(current_path, rendered_filename)
  • template_filename is the variable which contains the name of the reusable source file.
  • rendered_filename is the variable which contains the name of the output file which has been rendered trough Jinja2
  • render_vars contains the dictionary which will be passed to Jinja2 to render the source
  • script_path  contains the absolute path of the renderer.py script. E.g.
    • If the absolute path to renderer.py is /home/dummy/renderer.py then it would contain /home/dummy
  • template_file_path contains the absolute path to the source template file which will be passed to Jinja2
  • rendered_file_path contains the absolute path to the rendered file

 

Render the source template

Now we’re defining an environment which describes where to find the source template.

Defining a render environment
render_environment = jinja2.Environment(loader=jinja2.FileSystemLoader(template_file_path))

We’re using the jinja2.FileSystemLoader() as loader which takes the absolute path to the folder where the template is located.

 

To render the file and save the output into a variable we execute the render() method on render_environment and pass the render_vars dictionary.

Render variables into readable text
output_text = environment.get_template(template_filename).render(render_vars)

 

Putting it all together results in a static render script which prints out the rendered text.

Example - Print rendered Text
#!/usr/bin/env python3
import os
import jinja2

template_filename = "./template.html.j2"
rendered_filename = "rendered.html"
render_vars = {
    "greeting": "Oh hi!"
}

script_path = os.path.dirname(os.path.abspath(__file__))
template_file_path = os.path.join(script_path, template_filename)
rendered_file_path = os.path.join(script_path, rendered_filename)

environment = jinja2.Environment(loader=jinja2.FileSystemLoader(script_path))
output_text = environment.get_template(template_filename).render(render_vars)

print output_text
[email protected]: python3 /home/dummy/renderer.py
<h1>Oh hi!<h1>

 

Writing the output into an HTML file

To write the output into an HTML File you need to replace the last print line with the With-Statement and write the output_text into rendered_file_path.

Replacing print with fileopen
with open(rendered_file_path, "w") as result_file:
    result_file.write(output_text)

As soon as the With-Statement reaches its end, it automatically closes the file and destroys the result_file object. The end of the With-Statement is after the last indented line.

 

Further Ressources

Jinja2 Documentation: http://jinja.pocoo.org/docs/

Multithreading in Python 2.7

Multithreading in Python 2.7

Hey guys,

Today I’d like to tell you how to use multiple threads for heavy computing purposes.

For example, we (at Immowelt) needed Multithreading to speed up our static code analysis. The old process, based on bash, took up to 30 seconds each time the pre-commit Githook triggered. As soon as we used Multithreading the process executed within 5 seconds.

At first, we need to figure which packages we need to import. To use the most simple kind of multithreading we only need the ThreadPool Module which is part of multiprocessing.pool. I also need some other modules for demonstration purposes.

Import Modules; Start execution timer
from multiprocessing.pool import ThreadPool
import random, time, timeit

start_time = timeit.default_timer()

Then we define all needed variables and also the heavy calculating ( 😉 ) function.

Heavy Computing Function
def asyncProcess(threadNumber, threadRuntime):
    time.sleep(threadRuntime)
    return("%i: this Process ran %i seconds" % (threadNumber, threadRuntime))

Everything is ready to go, so we start the main function.

Main Function
if __name__ == '__main__':
    thread_count = 10 # Define the limit of concurrent running threads
    thread_pool = ThreadPool(processes=thread_count) # Define the thread pool to keep track of the sub processes
    known_threads = {}

    # Now we execute 10 parallel threads
    for i in range(0,10):
        known_threads[i] = thread_pool.apply_async(asyncProcess, args=(i,random.randint(0,10),))

    thread_pool.close() # After all threads started we close the pool
    thread_pool.join() # And wait until all threads are done

    # Getting the results of all threads
    for thread in known_threads:
        print known_threads[thread].get()

    print("Parent: this Process ran %s seconds" % str(timeit.default_timer() - start_time))

The Main Function cover the management of the Thread Pool, start all threads, wait until every child is done and gather the information of the child threads.

  • Thread Management (Line 2-4)
    • The limit of concurrent threads must be set. Be careful if you produce a heavy amount of network traffic or IOPS.
    • To spawn any child processes the thread pool must be defined
    • If you don’t have any fire-and-forget processes all children must be known
  • Start all threads (Line 7-8)
    • Here we start 10 Processes which wait 10 seconds and return the needed information
    • All children will be mapped in a dict object to gain their information after the thread pool is done
    • Very important: at line eight (the marked one) the last argument of args= must end with a comma-sign.
  • Wait until every child is done (Line 10-11)
    • The <pool>.close() method tells the pool object to stop accepting new child’s.
    • <pool>.join() blocks the parent until all child’s are done.
  • Gather computed information of the child threads (Line 14-15)
    • We iterate through the dict object and use the <child_thread>.get() method to print the return value generated by the children.

 

Putting it all together:

Whole Code
# Importing all needed modules
from multiprocessing.pool import ThreadPool
import random, time, timeit

# Starting timer for Parent measurement
start_time = timeit.default_timer()

# Define the function which will be executed within the ThreadPool
def asyncProcess(threadNumber, threadRuntime):
    time.sleep(threadRuntime)
    return("%i: this Process ran %i seconds" % (threadNumber, threadRuntime))

if __name__ == '__main__':
    thread_count = 10 # Define the limit of concurrent running threads
    thread_pool = ThreadPool(processes=thread_count) # Define the thread pool to keep track of the sub processes
    known_threads = {}

    # Now we execute 10 parallel threads
    for i in range(0,10):
        known_threads[i] = thread_pool.apply_async(asyncProcess, args=(i,random.randint(0,10),))

    thread_pool.close() # After all threads started we close the pool
    thread_pool.join() # And wait until all threads are done

    # Getting the results of all threads
    for thread in known_threads:
        print known_threads[thread].get()

    print("Parent: this Process ran %s seconds" % str(timeit.default_timer() - start_time))

Example Output

#: python multithreading.py
0: this Process ran 9 seconds
1: this Process ran 3 seconds
2: this Process ran 2 seconds
3: this Process ran 1 seconds
4: this Process ran 5 seconds
5: this Process ran 5 seconds
6: this Process ran 1 seconds
7: this Process ran 10 seconds
8: this Process ran 2 seconds
9: this Process ran 1 seconds
Parent: this Process ran 10.0648648739 seconds