Month: July 2017

Multithreading in Python 2.7

Multithreading in Python 2.7

Hey guys,

Today I’d like to tell you how to use multiple threads for heavy computing purposes.

For example, we (at Immowelt) needed Multithreading to speed up our static code analysis. The old process, based on bash, took up to 30 seconds each time the pre-commit Githook triggered. As soon as we used Multithreading the process executed within 5 seconds.

At first, we need to figure which packages we need to import. To use the most simple kind of multithreading we only need the ThreadPool Module which is part of multiprocessing.pool. I also need some other modules for demonstration purposes.

Import Modules; Start execution timer
from multiprocessing.pool import ThreadPool
import random, time, timeit

start_time = timeit.default_timer()

Then we define all needed variables and also the heavy calculating ( 😉 ) function.

Heavy Computing Function
def asyncProcess(threadNumber, threadRuntime):
    time.sleep(threadRuntime)
    return("%i: this Process ran %i seconds" % (threadNumber, threadRuntime))

Everything is ready to go, so we start the main function.

Main Function
if __name__ == '__main__':
    thread_count = 10 # Define the limit of concurrent running threads
    thread_pool = ThreadPool(processes=thread_count) # Define the thread pool to keep track of the sub processes
    known_threads = {}

    # Now we execute 10 parallel threads
    for i in range(0,10):
        known_threads[i] = thread_pool.apply_async(asyncProcess, args=(i,random.randint(0,10),))

    thread_pool.close() # After all threads started we close the pool
    thread_pool.join() # And wait until all threads are done

    # Getting the results of all threads
    for thread in known_threads:
        print known_threads[thread].get()

    print("Parent: this Process ran %s seconds" % str(timeit.default_timer() - start_time))

The Main Function cover the management of the Thread Pool, start all threads, wait until every child is done and gather the information of the child threads.

  • Thread Management (Line 2-4)
    • The limit of concurrent threads must be set. Be careful if you produce a heavy amount of network traffic or IOPS.
    • To spawn any child processes the thread pool must be defined
    • If you don’t have any fire-and-forget processes all children must be known
  • Start all threads (Line 7-8)
    • Here we start 10 Processes which wait 10 seconds and return the needed information
    • All children will be mapped in a dict object to gain their information after the thread pool is done
    • Very important: at line eight (the marked one) the last argument of args= must end with a comma-sign.
  • Wait until every child is done (Line 10-11)
    • The <pool>.close() method tells the pool object to stop accepting new child’s.
    • <pool>.join() blocks the parent until all child’s are done.
  • Gather computed information of the child threads (Line 14-15)
    • We iterate through the dict object and use the <child_thread>.get() method to print the return value generated by the children.

 

Putting it all together:

Whole Code
# Importing all needed modules
from multiprocessing.pool import ThreadPool
import random, time, timeit

# Starting timer for Parent measurement
start_time = timeit.default_timer()

# Define the function which will be executed within the ThreadPool
def asyncProcess(threadNumber, threadRuntime):
    time.sleep(threadRuntime)
    return("%i: this Process ran %i seconds" % (threadNumber, threadRuntime))

if __name__ == '__main__':
    thread_count = 10 # Define the limit of concurrent running threads
    thread_pool = ThreadPool(processes=thread_count) # Define the thread pool to keep track of the sub processes
    known_threads = {}

    # Now we execute 10 parallel threads
    for i in range(0,10):
        known_threads[i] = thread_pool.apply_async(asyncProcess, args=(i,random.randint(0,10),))

    thread_pool.close() # After all threads started we close the pool
    thread_pool.join() # And wait until all threads are done

    # Getting the results of all threads
    for thread in known_threads:
        print known_threads[thread].get()

    print("Parent: this Process ran %s seconds" % str(timeit.default_timer() - start_time))

Example Output

#: python multithreading.py
0: this Process ran 9 seconds
1: this Process ran 3 seconds
2: this Process ran 2 seconds
3: this Process ran 1 seconds
4: this Process ran 5 seconds
5: this Process ran 5 seconds
6: this Process ran 1 seconds
7: this Process ran 10 seconds
8: this Process ran 2 seconds
9: this Process ran 1 seconds
Parent: this Process ran 10.0648648739 seconds

Handcrafted St(r)eam Machine – not as easy as you think

Handcrafted St(r)eam Machine – not as easy as you think

Hey guys,

today I’d like to show you how I build my Steam Machine and why it isn’t as easy as it seems like.

I had the following conditions for the Steam Machine (which are no particularly high requirements):

  • Have to match into the closet
    • width: 50cm
    • height: 30cm
    • depth: 40cm
  • Most bang for bucks – upper limit 1000€

Hardware Parts

I end up choosing the following Parts for the Computing (Server) Part of my Setup:

Nr. Manufacturer Part Price in €
1 Corsair DIMM 16GB DDR4-2133 Kit, RAM 119,9
2 Samsung MZ-75E250B 250 GB, SSD 99,9
3 Fractal Design NODE 304, Case 74,9
4 Sharkoon SilentStorm SFX Gold 500W, PS 79,9
5 Intel Core i5 6500 4x 3.20GHz So.1151 BOX 202,3
6 Asus H110I-PLUS Intel H110 So.1151 Dual Channel DDR4 Mini-ITX Retail 70,34
7 Nvidia 6GB Inno3D GeForce GTX 1060 iChill X3 Aktiv PCIe 3.0 x16 259,00

As Clients, I used two Steam links located in the Bed and also Living Room. They work out of the Box and are able to customize with the Steam link SDK.

Now we miss two parts. Windows 10 disables the Cursor if no Mouse is connected. You are able to install any kind of software to emulate a mouse or plug in a very cheap one.

The second part is a virtual monitor which is very important. Some games try to get the Monitor configuration to set up the resolution. Either you try to install any kind of software to emulate a Monitor or you buy a HDMI Stick which emulates a monitor. You can use e.g. this Model.

Software Part

My Server will be running on Windows 10 because most games in Steam are compatible with Windows. To keep control of the server I use Teamviewer.

Windows 10 need some tweaks to work properly as streaming Machine because e.g. the login keeps appearing after reboot so that Steam isn’t able to start.

Disabling Login Screen on boot up

  • Why?
    • As soon as the Steam machine boot up, steam cannot start because the computer is still locked.
  • How?
    • a. Press [WINDOWS] Button or open start menu
    • b. Enter “netplwiz”
    • c. Remove the checkmark of “Users must enter a username and password to use this computer.”
    • d. Click OK to keep the option

Disabling Lock Screen on Wake on Lan

  • Why?
    • After Waking up the Machine it keeps stuck in a lock screen.
  • How?
    • a. Press [WINDOWS] + [X]
    • b. Select “Command Prompt (admin)”
    • c. Enter the following command
      • powercfg /SETACVALUEINDEX SCHEME_CURRENT SUB_NONE CONSOLELOCK 0

Disabling UAC (User Account Control)

  • Why?
    • The Steam link is not able to work through the UAC when e.g. installing a program.
  • How?
    • First Method:
      • a. Press [WINDOWS] + [X]
      • b. Go to “Control Panel\User Accounts and Family Safety\User Accounts”
      • c. Click on “Change User Account Control settings”
      • d. Move the slider to the bottom to “Never Notify”
      • e. Click ok
    • Second Method:
      • a. Open the Start Menu
      • b. enter ua c and hit [ENTER]
      • c. Go ahead with “First Method” at (d.)
    • Third Method:
      • a. Open Registry Editor (open Start Menu; enter Regedit; hit [ENTER]
      • b. Navigate to “HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System”
      • c. Modify the value of the “EnableLUA” DWORD and set it to 0

Putting all together

Does it even perform?

To perform any kind of performance tests I always use Novabench because it’s free with all features included. Here are my results.

Yes. It does! This score is really good in relation to the total costs.

Brief overview of my Homelab

Brief overview of my Homelab

Hey guys,

today I like to tell and also show you something about my home lab including my well-known, not done yet, security system and the not so smart ‘Smart Home’ which obviously becomes smarter as time passes by.

Infrastructure

The infrastructure consists of a Fritzbox Router (which I received from my Internet Service Provider) with a 1GB connection to a Cisco Tier 2 switch (the core switch). Bed- and Livingroom is also equipped with a 1GB switch (of course not cisco :-)). The switches are connected with CAT 7 cables which are jailed into a cable channel.

Security

As I told before my security system consists of two hardware components: an IP Cam and a cable connected motion detector. No siren at all. The motion detector is connected to my Crestron (will be explained later) Smarthome Base into an I/O Port. As soon as the motion detector detects motion and the Security system is armed, a Raspberry Pi receives a xinetd command and executes a script which takes a picture, sends it as mail, and records a video afterward which will be uploaded to my Dropbox account.

Smart Home

My Smarthome stuff contains everything. It starts from Crestron (a commercial solution) and ends with OpenHAB2 (a opensource solution). I started with Crestron as my apprenticeship begun and it only controls my current security system and all infrared devices. OpenHAB2 is way easier and quite cheap to implement different smart home techniques like Z-Wave or Mysensors. At the moment OpenHAB2 control all Z-Wave Devices and it started to take over the infrared devices in the Bedroom.

Where the way leads us to?

I want to completely redesign my security system and exchange Crestron with OpenHAB2.
Appendix

Crestron -> https://www.crestron.com/
OpenHAB2 -> http://docs.openhab.org/
Mysensors -> https://www.mysensors.org/

 

Thanks for reading. See you soon!

 

via Jason Brown and Shawn Douglas (Creative Commons Attribution-NonCommercial license)