Daniel Arbuckle’s Mastering Python

Daniel Arbuckle's Mastering Python

                   

Build powerful Python applications

                   

Daniel Arbuckle

 

BIRMINGHAM - MUMBAI

< html PUBLIC -//W3C//DTD HTML 4.0 Transitional//EN http://www.w3.org/TR/REC-html40/loose.dtd>

Daniel Arbuckle's Mastering Python

Copyright © 2017 Packt Publishing  

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: June 2017

Production reference: 1300617

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham 

B3 2PB, UK.

ISBN 978-1-78728-369-5

Credits

  

About the Author

Daniel Arbuckle gained his PhD in Computer Science from the University of Southern California. He has published numerous papers along with several books and video courses, and he is both a teacher of computer science and a professional programmer.

www.PacktPub.com

For support files and downloads related to your book, please visit www.PacktPub.com. Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.comand as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at service@packtpub.com for more details. At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.

https://www.packtpub.com/mapt

Get the most in-demand software skills with Mapt. Mapt gives you full access to all Packt books and video courses, as well as industry-leading tools to help you plan your personal development and advance your career.

Why subscribe?

Fully searchable across every book published by Packt

Copy and paste, print, and bookmark content

On demand and accessible via a web browser

Customer Feedback

Thanks for purchasing this Packt book. At Packt, quality is at the heart of our editorial process. To help us improve, please leave us an honest review on this book's Amazon page at www.amazon.com/dp/1787283690.

If you'd like to join our team of regular reviewers, you can email us at customerreviews@packtpub.com. We award our regular reviewers with free eBooks and videos in exchange for their valuable feedback. Help us be relentless in improving our products!

Table of Contents

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Errata

Piracy

Questions

Python Primer

Python basic syntax and block structure

Basic building blocks

Functions

Variables

Expressions

Classes

Flow control statements

Indentation

Python's built-in data structures and comprehensions

Dictionaries

List

Tuple

Set

Comprehension

First-class functions and classes

The defaultdict class

Attributes

The standard library

Different types of packages

What's new in modern Python

The changes in the syntactic

Changes in packages

Other changes in Python packages

Summary

Setting Up

Downloading and installing Python

Choosing a suitable version

Installing Python

Using the command line and the interactive shell

Opening a command-line window

Python interactive shell

Installing packages with pip

The pip tool for packages

Managing installed packages

Finding packages in the Python Package Index

Using keywords

Using Package Index

Searching the Package Index with pip

Legalities and licenses of the Python Package Index

Summary

Making a Package

Creating an empty package

Turning a regular folder into a package

Importing all package modules

Adding modules to the package

Module loading with namespace packages

The Package structure and interface

Accessing code from other modules

Importing a cyclic dependency

Resolving attribute errors raised due to cyclic dependencies

Adding static data files to the package

Summary

Basic Best Practices

PEP 8 and writing readable code

PEP 8 — guidelines for Python code

Code indentation

Formatting recommendations

Naming conventions

Using version control

Initializing Git

Committing the changes in Git

Undoing the changes

Branches

Merging codes

The mergetool command

The pull command

Using venv to create a stable and isolated work area

Creating a virtual environment

Activating a virtual environment

pip in virtual environments

Getting the most out of docstrings

PEP 257 and docutils

Sphinx

Turning docstrings into HTML

Using doctest to test documentation examples

Testing examples using doctest

What it means when a code example fails

Summary

Making a Command-Line Utility

Making a package executable via Python -m

Pipeline program

Handling command-line arguments with argparse

Creating an ArgumentParser object

Setting the name of argument

nargs

Python tools to interact with the user

Python's built-in functions - print and input

The getpass package

The pprint package

The cmd class

The Pipeline user interface

Executing other programs with subprocess

Subprocess and its variants

Using the Popen subprocess

The PIPE constant

The wait method

Finishing up our code example

Setting up a shell script or batch file to launch the program

Creating launches for our program

Summary

Parallel Processing

Using the concurrent.futures package

The concurrent.futures module

Calling ProcessPoolExecutor

Using the map method

Using the submit method

The done and result methods

The wait and as_completed functions

The add done callback function

The cancel method

Using the multiprocessing packages

Process class in the multiprocessing module

Queues

Pipes

Manager

The lock object

The event object

The condition object

The semaphore object

Summary

Coroutines and Asynchronous I/O

The difference between asynchronous processing and parallel processing

Multithreading is not good for servers

Cooperative coroutine scheduler versus coroutine

Python coroutines

The coroutine scheduler

Using the asyncio event loop and coroutine scheduler

Creating a coroutine

The asyncio scheduler - event_loop

ensure_future

The run_forever/run_until_complete methods

Closing event_loop

Awaiting data availability

asyncio's future objects

Asynchronous iterations

Synchronizing multiple tasks

Synchronization primitives

The wait coroutine

The wait_for coroutine

The gather coroutine

The asyncio Queue class

Queue types

Communicating across the network

Creating a simple client in asyncio

Creating a simple server in asyncio

Handling client disconnections

Summary

Metaprogramming

Using function decorators

Using the @ syntax in a function decorator

Global decorator - @staticmethod

Attributes

Enclosing the function in a wrapper

The @wraps decorator

The only function

Function annotations

Function annotation syntax

Accessing annotation data

The @no_type_check decorator

Annotations as input to function decorators

Keyword arguments

Inspecting the package signature function

Class decorators

Modifying class attributes

The factory function

The factory_constructed function

Class definitions

Metaclasses

What can we do with a metaclass?

The __prepare__method

The __new__ method

Context managers

Defining a context manager as a generator

Adding context manager behavior to a class

Synchronous-coroutine-based context managers

Creating an asynchronous-coroutine-based context manager

Descriptors

Using @property to create a descriptor

Writing descriptors as classes

Summary

Unit Testing

Understanding the principle of unit testing

What is a unit test?

Using the unittest package

Structuring a test file

assert methods

Comparing what happens to what should happen in unit tests

Using unittest.mock

What is a mock object?

Preconfiguring mock objects

assert methods of mock objects

The unittest.mock patch function

Using unittest's test discovery

Unittest's discovery tool

Command-line options in unit test discovery

Using nose for unified test discovery and reporting

Running our tests with nose

The cover-package option

Testing multiple worker processes

Summary

Reactive Programming

The concept of reactive programming

Building a simple reactive programming framework

Observers

Observables

Emitting events

Building the observable sequence

Illustrating a stream of animal events

Composing an observable sequence

Using the reactive extensions for Python (RxPY)

Translating our zoo demo into Rx

Observable factory methods

Explaining the observable sequence of events

Creating an asyncio scheduler

Combining and processing observable sequences

Miscellaneous observable factory methods

The Observable.create method

The Observable.select_many method

Empty, return_value, and from_iterable factory methods

The where factory method

Summary

Microservices

Microservices and the advantages of process isolation

Advantages of the microservice architecture

Applying the microservice architecture to web servers

Building high-level microservices with Flask

Installing Flask

Creating endpoints for a RESTful API in Flask

Building a microservice to maintain a database

Making Flask handle a request

Running and connecting to our microservice using Flask

Test running the microservice

Building high-level microservices with nameko

Installing nameko

Running and connecting a microservice using nameko

Things to know before using nameko

Interacting with our microservice

Interacting with a microservice manually using the nameko shell

Interacting with a microservice by creating another microservice

Summary

Extension Modules and Compiled Code

Advantages and disadvantages of compiled code

The downsides of compiled code

Accessing a dynamic library using ctypes

Locating and linking a dynamic library

Accessing functions defined in the library

Assigning attributes to a function

Using a pointer as a parameter of a function

Providing a function signature

Providing data structure layouts

Interfacing with C code using Cython

Working with Cython

Additional import methods in Cython

Writing extension modules in Cython

Methods to increase the execution speed of Python code

Using cpdef in a Cython class

Compiling an extension module in Python

Summary

Preface

Welcome to Daniel Arbuckle's Mastering Python. Python is a member of the C family of languages, such as C++ and Java; however, Python is more of a distant cousin, as the designers of Python are quite happy to do things in a different way if it seems better. Therefore, if you're familiar with C or one of its descendant languages, you'll find Python relatively familiar.

The goal of this book is to help you move from being a Python newcomer to being able to use an assortment of advanced techniques. Along with C, C++, and Java, Python is one of the languages that everybody needs to know well. Python is, in my humble opinion, the best all-round language available and it's a whole lot of fun too!

We will progress roughly from entry-level to an advanced level, but for the most part, each chapter is independent of the others. Feel free to skip around to the things that you need to learn, or work through the course in order to get up to speed. So, let's get you there.

What this book covers

Chapter 1, Python Primer, is a quick primer on the Python language's syntax and semantics.

Chapter 2, Setting Up, is about getting the Python runtime installed and usable.

Chapter 3, Making a Package, shows how to create a Python source code package.

Chapter 4, Basic Best Practices, covers several of the best practices, including source code formatting rules and the use of tools such as version control in virtual environments.

Chapter 5, Making a Command-Line Utility, explains how to create a complete text-mode utility program.

Chapter 6, Parallel Processing, shows how to improve the performance of CPU-bound programs in parallel processing.

Chapter 7, Coroutines and Asynchronous I/O, explains how to improve the performance of I/O bound programs using asynchronous I/O.

Chapter 8, Metaprogramming, covers several different ways in which Python syntax or semantics can be controlled programmatically from within our own source code.

Chapter 9, Unit Testing, talks about automated unit testing and test-driven development.

Chapter 10, Reactive Programming, is about reactive programming and the RxPY framework.

Chapter 11, Microservices, is about creating microservices.

Chapter 12, Extension Modules and Compiled Code, talks about linking Python code with system-level code written in C.

What you need for this book

You will need Ubuntu 16.04 and Python (version 3.6). You can also run the code examples on Windows and macOS. 

You may opt to use VirtualBox as well to test the code in the book. 

Who this book is for

If you are a programmer and are familiar with the basics of Python and you want to broaden your knowledge base to develop projects better and faster, this book is for you. Even if you are not familiar with Python, our book starts with the basics and takes you on a journey to become an expert in the technology. 

Conventions

In this book, you will find a number of text styles that distinguish between different kinds of information. Here are some examples of these styles and an explanation of their meaning. Code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles are shown as follows: Using a name variable to look up the stored value is an expression; so is running a function.

A block of code is set as follows:

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:

Any command-line input or output is written as follows:

New terms and important words are shown in bold. Words that you see on the screen, for example, in menus or dialog boxes, appear in the text like this: Click on Environments Variables....

Warnings or important notes appear in a box like this.

Tips and tricks appear like this.

Reader feedback

Feedback from our readers is always welcome. Let us know what you think about this book-what you liked or disliked. Reader feedback is important for us as it helps us develop titles that you will really get the most out of. To send us general feedback, simply email feedback@packtpub.com, and mention the book's title in the subject of your message. If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide at www.packtpub.com/authors.

Customer support

Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.

Downloading the example code

You can download the example code files for this book from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support  and register to have the files emailed directly to you. You can download the code files by following these steps:

Log in or register to our website using your email address and password.

Hover the mouse pointer on the SUPPORT tab at the top.

Click on Code Downloads & Errata.

Enter the name of the book in the Search box.

Select the book for which you're looking to download the code files.

Choose from the drop-down menu where you purchased this book from.

Click on Code Download.

Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:

WinRAR / 7-Zip for Windows

Zipeg / iZip / UnRarX for Mac

7-Zip / PeaZip for Linux

The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Daniel-Arbuckles-Mastering-Python. We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Errata

Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books-maybe a mistake in the text or the code-we would be grateful if you could report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by