DS 231: Introduction to Data Science Programming

Part 1: Open data

Open data is data that has been made publicly available and may be used, reused, built on and shared. It belongs to the wider “open movement” (open source software, open hardware, open access journals). Open licenses use copyleft instead of copyright. Governments release open government data to promote transparency and accountability, and volunteers use it to build solutions to social problems.

Saudi Open Data Portal (data.gov.sa)

• A national initiative for a public data hub that supports transparency, e-participation and innovation.
• Publishes datasets from ministries and government agencies in an open format, giving the public one central access point to find, download and use them.
• Benefits: understand how agencies work, evaluate their performance, make informed decisions about policy, and build research, reports, web and mobile apps.
• Facts: 6,544 datasets (August 2022), 147 publishers, use cases available, you can request a dataset. The open data license lets users distribute, transform and build on the data, with the source credited.

Part 2: Why Python?

In 2017 Python became the most popular language in the world (IEEE Spectrum). Three reasons:

1. It is relatively easy to learn.
2. Everything you need is free.
3. It has more ready-made tools for data science, machine learning, AI and robotics than most languages.

Choosing the right Python

Go to python.org. It recommends the most current stable version. Use that one.

Tools for success

• An editor lets you type code; an interpreter runs it. You need both.
• Anaconda is a complete Python development environment with a friendly graphical interface, often called a data science platform because many of its packages are data-science oriented. It also comes with VS Code.
• Install: go to anaconda.com/download, download the free version, follow the prompts, and choose to install Microsoft VS Code when asked.
• Anaconda Navigator lets you move between features and launch what you want.
• Open VS Code from Anaconda with its Launch button. Check the Extensions icon (puzzle piece): you should see Anaconda Extension Pack, Python and YAML.
• Jupyter Notebook is another popular tool for writing Python. It supports Julia, Python and R, is free with Anaconda, and is often used to share code online.

Interactive mode in the VS Code terminal

1. Open Anaconda Navigator and Launch VS Code.
2. If the Terminal pane is hidden, choose View → Terminal.
3. Check your version at the operating system prompt (space before the first hyphen, no other spaces):

python --version

You should see something like Python 3.x.x. Now start the interpreter:

python
>>>

The >>> prompt means you are inside the Python interpreter. Type a command and press Enter. A spelling mistake gives an error message.

Built-in help

>>> help()
help> keywords

• help> means you left the interpreter (>>>) and are in the help area. Type a module, keyword or topic name to get help.
• keywords lists the words with special meaning in Python.
• Leave help with q (quit) or Ctrl+Z, then leave Python with exit().
• Online help: YouTube for videos, Stack Overflow for questions, plus search engines.

Workspace, folder and your first file

• A VS Code workspace is your development environment: the Python interpreter you use plus any extensions you add. Store it anywhere.
• Create a folder for all your Python code and associate it with the workspace, so you always use the right interpreter and settings.
• Each Python file is a plain text file with the .py extension. The VS Code editor is a code editor.

Your first program, Hello World:

print("Hello World")

Hello World

• Saving: VS Code does not save automatically. Save often, or turn on File → Auto Save (a check mark means it is on).
• Running: right-click the file name and choose Run Python File in Terminal. The terminal shows the output and a new prompt.

Simple debugging

VS Code shows errors before you run: red file and folder names, a red error count, a red circled X total at the bottom left, and a wavy red underline on the bad code.

VS Code also has a built-in debugger for more complex programs.

Jupyter Notebook

• Notebooks are saved with the .ipynb extension. Save with File → Save and Checkpoint.
• A notebook is made of cells. A Code cell holds Python; the output appears below it.
• Run a cell with Alt+Enter (Windows), Option+Enter (Mac) or the Run button. Only the cell with the cursor runs; the double-triangle icon runs all cells.
• Markdown cells hold formatted text, pictures and video. Markdown is like a greatly simplified HTML. Insert a cell with Insert → Insert Cell Below.
• To reopen, launch Jupyter Notebook from Anaconda and click the file.

The Zen of Python

Many languages focus on what the computer does. Python is built around how humans think, work and communicate. Python code should be more human-readable than machine-readable.

Object-oriented programming (OOP)

OOP mimics the real world: it consists of objects with properties and methods (actions). Think of a car:

Python is very much object-oriented, but first learn the core language so you know how to use other people’s objects.

Why indentation counts, big time

Using Python modules

Python has a simple, clean core language, plus hundreds of free modules written and tested by other people for specific jobs (science, AI, dates and times). You import a module and use it as its documentation says. Newer versions may exist, but if your version works you need not upgrade.

import random
answer = random.randint(1, 8)   # random whole number from 1 to 8

Core Python has no random number generator built in, so import random brings it in.

Import syntax

import modulename [as alias]

• Written in lowercase exactly as shown (case-sensitive).
• Italic words are placeholders you replace with your own.
• Anything in [square brackets] is optional, and you never type the brackets.
• Put import statements first in the file.

Aliases

An alias is a nickname after the word as. It saves typing when module names are long.

import random as rnd
answer = rnd.randint(1, 8)

Comments

A comment is text that does nothing when the program runs. It explains the code to teammates and to your future self. Two ways:

# This is a Python comment (starts with a pound sign)

"""This is a multiline comment.
It is sometimes called a docstring.
It starts and ends with three double quotation marks."""

Data types

Computers can do arithmetic on numbers, not on words.

What is a valid number?

It starts with a digit, a decimal point or a minus sign, and has at most one decimal point, with no letters, spaces, $ or commas.

Quotes in strings

"Mary's dog said Woof"            # apostrophe inside: use double quotes
'The dog of Mary said "Woof".'     # double quotes inside: use single quotes
x = True                           # Boolean, no quotes, capital T

Operators

Variables

A variable is a placeholder for information that may change. The = sign is the assignment operator: it assigns the value on the right to the variable on the left.

x = 10                       # store 10 in the variable x
quantity = 3
unit_price = 6.5
extended_price = quantity * unit_price
print(extended_price)

19.5

Rules for variable names

• Must start with a letter or an underscore (_).
• After the first character you can use letters, numbers or underscores.
• Case-sensitive: Price and price are different.
• No quotation marks inside the name.
• PEP 8 style: lowercase letters with underscores between words (unit_price). Use meaningful names.

Syntax

Syntax is the proper order and format of code, as important as in human language. If you break it, Python shows a SyntaxError. A line of code ends with a line break or a semicolon, and both of these run the same:

first_name = "Alan"
last_name = "Simpson"
print(first_name, last_name)

first_name = "Alan"; last_name = "Simpson"; print(first_name, last_name)

The exercises in this module (type, save, run, change, save again, run again) are what you do in any software development, so practise them until they feel natural.

Calling functions

variablename = functionname(param[, param])

Most functions return a value, so you store it in a variable. The values inside the parentheses are parameters (arguments). abs() takes one number; round() takes a number and then the number of decimal places.

z = -999.9999
print(abs(z))
print(int(abs(z)))
print(round(3.14159265, 4))
print(type(3.14), type(128), type(str(5)))

999.9999
999
3.1416
<class 'float'> <class 'int'> <class 'str'>

The math module

More functions live in the math module. Import it first, then write the module name, a dot, and the function name.

import math
print(math.sqrt(81))
print(math.factorial(7))
print(math.floor(-23234.5454))
print(math.radians(45))

9.0
5040
-23235
0.7853981633974483

Formatting numbers with f-strings

Put a lowercase f (or F) right before the opening quote. The text inside is the literal part (shown exactly). Anything in curly braces {} is the expression part: a variable or a calculation that gets filled in.

quantity = 3
unit_price = 499.90
print(f"Subtotal: ${quantity * unit_price:,.2f}")

Subtotal: $1,499.70

• The format string starts with a colon and sits inside the closing brace, right after the variable or value.
• The $ is in the literal part, outside the braces, so alignment does not move it.
• Multiline output: use \n in the literal part (not inside braces), or put the f-string in triple quotes and break lines where you want them.

user1 = "Alberto"; user2 = "Babs"; user3 = "Carlos"
print(f"{user1} \n{user2} \n{user3}")

Alberto 
Babs 
Carlos

Binary, octal and hexadecimal

To convert to decimal you need no function. Just print the number: print(0b11111111), print(0o377) and print(0xff) all show 255.

Manipulating strings

Concatenation and length

Join strings with + (concatenation). Python does not add spaces for you, so add " " yourself. len() counts characters, including spaces.

full_name = "Alan" + " " + "C" + ". " + "Simpson"
print(full_name)
print(len(""), len(" "), len("A B C"))

Alan C. Simpson
0 1 5

String operators

Characters are compared by their ASCII numbers: space is 32, A to Z are 65 to 90, and a to z are 97 to 122. That is why max("Abc") is c.

String methods

Dates and times

There is no built-in date type. Import the module (an alias is common):

import datetime as dt

Working with dates

today = dt.date.today()               # from the computer's clock
last_of_teens = dt.date(2019, 12, 31)    # year, month, day, in that order
print(last_of_teens.month, last_of_teens.day, last_of_teens.year)

12 31 2019

Working with times and datetimes

right_now = dt.datetime.now()
print(right_now)

2019-11-19 14:03:07.525975

Formatting dates with strftime directives

Example: the format %A %B %d is day number %j of %Y gives Saturday June 01 is day number 152 of 2019.

Calculating timespans with timedelta

A timedelta object is created automatically when you subtract two dates, times or datetimes. It measures “how long”, not “when”.

today = dt.date.today()
birthdate = dt.date(2000, 1, 31)
delta_age = today - birthdate          # a timedelta
days_old = delta_age.days
years = days_old // 365                # floor division gives whole years
months = (days_old % 365) // 30        # leftover days, about 30 days per month
print(f"You are {years} years and {months} months old.")

You are 18 years and 9 months old.

Time zones

The time from your system clock (now()) is in your own zone but does not say which. Compare it with UTC (utcnow()) and subtract to find your offset. To get the time in specific zones, use gettz from dateutil.tz:

import datetime as dt
from dateutil.tz import gettz
utc = dt.datetime.now(gettz('Etc/UTC'))
est = dt.datetime.now(gettz('America/New_York'))
print(f"{est:%A %D %I:%M %p %Z}")

Operators for making decisions

• Relational (comparison) operators < <= > >= == != compare two items to see how they are related.
• Logical (Boolean) operators and, or, not combine several comparisons before a final decision.

The if statement

The condition is followed by a colon. Indent the lines that belong to the if. They run only if the condition is true. The first un-indented line after the block runs no matter what.

total = 100
sales_tax_rate = 0.065
taxable = True
if taxable:
    print(f"Subtotal : ${total:.2f}")
    sales_tax = total * sales_tax_rate
    print(f"Sales Tax: ${sales_tax:.2f}")
    total = total + sales_tax
print(f"Total    : ${total:.2f}")

Subtotal : $100.00
Sales Tax: $6.50
Total    : $106.50

With taxable = False, all four indented lines are skipped and the total stays $100.00.

Adding else

Lines indented under else: run only if the condition was not true.

import datetime as dt
now = dt.datetime.now()
if now.hour < 12:
    print("Good morning")
else:
    print("Good afternoon")
print("I hope you are doing well!")

Good morning
I hope you are doing well!

But at 11 at night, “Good afternoon” is wrong. That is where elif comes in.

Handling many cases with elif

An if can have any number of elif conditions. A final else is optional and runs only if the if and all the previous elifs were false.

light_color = "yellow"
if light_color == "green":
    print("Go")
elif light_color == "red":
    print("Stop")
else:
    print("Proceed with caution")
print("This code executes no matter what")

Proceed with caution
This code executes no matter what

Note: the contents list for Module 12 also mentions repeating a process with for and looping with while, but the slides in this file cover only Part 1 (decisions with if).