10 Development Tips For Python [Part-2]

Note: This article was followed by article 10 Development Tips For Python [Part-1]

Hashtags: #python #PythonProgramming #Coding #Programming #PythonDevelopment

How to quickly calculate function running time


To calculate the running time of a function, you might do it like this:

import time

start = time.time()

# run the function

end = time.time()

print(end-start)

You see, you have written a few lines of code to calculate the running time of the function.

Is there a way to calculate this running time more conveniently?

The answer is yes.

There is a built-in module called Timeit

Use it with just one line of code.

import time

import timeit

def run_sleep(second):

print(second)

time.sleep(second)

# Just use this line

print(timeit.timeit(lambda :run_sleep(2), number=5))

The results are as follows:

2

2

2

2

2

10.020059824

Use the built-in caching mechanism to improve efficiency


Caching is a processing method that saves quantitative data to meet subsequent acquisition needs, and aims to speed up data acquisition.


The data generation process may require operations such as calculation, regularization, and remote acquisition. If the same piece of data needs to be used multiple times, it would be a waste of time to regenerate each time. Therefore, if the data obtained by operations such as calculation or remote request is cached, the subsequent data acquisition requirements will be accelerated.


In order to achieve this requirement, Python 3.2+ provides us with a mechanism that can be easily implemented without requiring you to write such logic codes.


This mechanism is implemented in the lru_cache decorator in the functool module.

@functools.lru_cache(maxsize=None, typed=False)

Parameter interpretation:


(1) maxsize: How many call results of this function can be cached at most. If it is None, there is no limit. When set to a power of 2, the performance is best


(2) typed: If True, calls with different parameter types will be cached separately.

For example:

from functools import lru_cache

@lru_cache(None)

def add(x, y):

print("calculating: %s + %s" % (x, y))

return x + y

print(add(1, 2))

print(add(1, 2))

print(add(2, 3))

The output is as follows, you can see that the second call does not actually execute the function body, but directly returns the result in the cache.

calculating: 1 + 2

3

3

calculating: 2 + 3

5

The following is a classic Fibonacci sequence. When you specify a larger n, there will be a lot of repeated calculations.

def fib(n):

if n < 2:

return n

return fib(n - 2) + fib(n - 1)

The timeit introduced in point 6 can now be used to test how much efficiency can be improved.

Without lru_cache, the running time is 31 seconds.

import timeit

def fib(n):

if n < 2:

return n

return fib(n - 2) + fib(n - 1)

print(timeit.timeit(lambda :fib(40), number=1))

# output: 31.2725698948

After using lru_cache, the running speed is too fast, so I adjusted the value of n from 30 to 500, but even so, the running time is only 0.0004 seconds. The speed increase is very significant.

import timeit

from functools import lru_cache

@lru_cache(None)

def fib(n):

if n < 2:

return n

return fib(n - 2) + fib(n - 1)

print(timeit.timeit(lambda :fib(500), number=1))

# output: 0.0004921059880871326

Tips for executing code before program exit


Using the built-in module "atexit", you can easily register and exit functions.


No matter where you cause the program to crash, the functions you registered will be executed.


If the clean() function has parameters, you can directly call atexit.register(clean_1, parameter 1, parameter 2, parameter 3='xxx') without using the decorator.


Maybe you have other ways to deal with this kind of demand, but it's definitely more elegant than not using atexit, it's convenient, and it's easy to extend.


But using atexit still has some limitations, such as:


  • If the program is killed by a system signal that you have not processed, the registered function cannot be executed normally.

  • If a serious Python internal error occurs, the function you registered cannot execute normally.

  • If you manually call os._exit(), the function you registered cannot execute normally.


Implement deferred calls like defer


There is a delayed call mechanism in Golang, the keyword is defer, the example is as follows:

import "fmt"

func myfunc() {

fmt.Println("B")

}

func main() {

defer myfunc()

fmt.Println("A")

}


The output is as follows, the call of myfunc will be completed one step before the function returns, even if you write the call of myfunc in the first line of the function, this is a delayed call.

A

B

So is there such a mechanism in Python?

Of course there are, but it is not as simple as Golang.

In Python, you can use the context manager to achieve this effect

import contextlib

def callback():

print('B')

with contextlib.ExitStack() as stack:

stack.callback(callback)

print('A')

The output is as follows:

A

B


How to read gigabytes of files in a stream


Use "with...open..." to read data from a file, which is an operation that all Python developers are very familiar with.


But if you use it improperly, it will also cause a lot of trouble.


For example, when you use the read function, Python will actually load all the contents of the file into memory at once. If the file has 10 G or more, then your computer will consume a huge amount of memory.

# One-time read

with open("big_file.txt", "r") as fp:

Content = fp.read()

For this problem, you might think of using readline as a generator to return line by line.

def read_from_file(filename):

with open(filename, "r") as fp:

yield fp.readline()

But if the content of this file is one line, 10 Gs per line, in fact, you will still read all the content at once.

The most elegant solution is to specify that only a fixed size of content is read each time when using the read method. For example, in the following code, only 8kb is returned each time.

def read_from_file(filename, block_size = 1024 * 8):

with open(filename, "r") as fp:

while True:

chunk = fp.read(block_size)

if not chunk:

break

yield chunk

The above code is functionally okay, but the code still looks bloated.

With the help of partial function and iter function, the code can be optimized:

from functools import partial

def read_from_file(filename, block_size = 1024 * 8):

with open(filename, "r") as fp:

for chunk in iter(partial(fp.read, block_size), ""):

yield chunk

About

Launched in 2016 as 591Lab International and locally in China known as “WUQIUYAO Tech. Ltd” we are committed to offering our clients excellent experience on ISACA, PMI, Cisco and Huawei examination preparatory services. We focus strongly on popular exams, and exam preparations services. We provide our customers with the complete training needed to earn the best scores for their respective Management and IT career certifications. We have a huge list of satisfied customers with top grades to back up all the claims we make.

Quick Links

Contact

This material is not sponsored by, endorsed by, or affiliated with Cisco Systems, Inc & Huawei Technologies Co., Ltd. Cisco Certified Internetworking Engineer, the Cisco Systems logo and the CCIE™ logo are trademarks or registered trademarks of Cisco Systems, Inc. in the United States and certain other countries.Huawei Certified Internetwork Expert, the Huawei logo and the HCIE™ logo are trademarks or registered trademarks of Huawei Technologies Co., Ltd . in China and certain other countries All other trademarks are trademarks of their respective owners. 

© Copyright 591Lab 2020. All Rights Reserved.