Python Comprehensions & Iterators

1. List Comprehensions

Purpose: Create lists in a concise, readable way using a single line of code.

Basic Syntax:

python

[expression for item in iterable]

Basic Examples:

python

# Traditional way
squares = []
for x in range(5):
    squares.append(x ** 2)
print(squares)  # [0, 1, 4, 9, 16]

# List comprehension way
squares = [x ** 2 for x in range(5)]
print(squares)  # [0, 1, 4, 9, 16]

# More examples
numbers = [1, 2, 3, 4, 5]
doubled = [x * 2 for x in numbers]
print(doubled)  # [2, 4, 6, 8, 10]

# With strings
words = ["hello", "world", "python"]
lengths = [len(word) for word in words]
print(lengths)  # [5, 5, 6]

List Comprehensions with Conditions:

python

# Basic condition
even_squares = [x ** 2 for x in range(10) if x % 2 == 0]
print(even_squares)  # [0, 4, 16, 36, 64]

# Multiple conditions
numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
filtered = [x for x in numbers if x > 3 and x % 2 == 0]
print(filtered)  # [4, 6, 8, 10]

# If-else in comprehension
categorized = ["even" if x % 2 == 0 else "odd" for x in range(5)]
print(categorized)  # ['even', 'odd', 'even', 'odd', 'even']

Nested List Comprehensions:

python

# Flatten a 2D list
matrix = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
flattened = [num for row in matrix for num in row]
print(flattened)  # [1, 2, 3, 4, 5, 6, 7, 8, 9]

# Create a matrix
matrix = [[i * 3 + j for j in range(3)] for i in range(3)]
print(matrix)  # [[0, 1, 2], [3, 4, 5], [6, 7, 8]]

2. Dictionary Comprehensions

Purpose: Create dictionaries in a concise way.

Basic Syntax:

python

{key_expression: value_expression for item in iterable}

Basic Examples:

python

# Create dictionary from list
numbers = [1, 2, 3, 4, 5]
squares_dict = {x: x ** 2 for x in numbers}
print(squares_dict)  # {1: 1, 2: 4, 3: 9, 4: 16, 5: 25}

# From two lists
keys = ['a', 'b', 'c']
values = [1, 2, 3]
combined = {k: v for k, v in zip(keys, values)}
print(combined)  # {'a': 1, 'b': 2, 'c': 3}

# Transform existing dictionary
original = {'a': 1, 'b': 2, 'c': 3}
doubled = {k: v * 2 for k, v in original.items()}
print(doubled)  # {'a': 2, 'b': 4, 'c': 6}

Dictionary Comprehensions with Conditions:

python

# Filter dictionary items
scores = {'Alice': 85, 'Bob': 92, 'Charlie': 78, 'Diana': 95}
high_scores = {name: score for name, score in scores.items() if score >= 90}
print(high_scores)  # {'Bob': 92, 'Diana': 95}

# Transform keys and values
data = {'temp1': 25, 'temp2': 30, 'pressure1': 100}
cleaned = {k.replace('temp', 'temperature'): v for k, v in data.items()}
print(cleaned)  # {'temperature1': 25, 'temperature2': 30, 'pressure1': 100}

# Switch keys and values with condition
original = {'a': 1, 'b': 2, 'c': 3}
swapped = {v: k for k, v in original.items() if v > 1}
print(swapped)  # {2: 'b', 3: 'c'}

3. Set Comprehensions

Purpose: Create sets in a concise way (automatically removes duplicates).

Basic Syntax:

python

{expression for item in iterable}

Basic Examples:

python

# Create set from list (removes duplicates)
numbers = [1, 2, 2, 3, 3, 3, 4, 4, 4, 4]
unique_numbers = {x for x in numbers}
print(unique_numbers)  # {1, 2, 3, 4}

# Mathematical operations
squares_set = {x ** 2 for x in range(5)}
print(squares_set)  # {0, 1, 4, 9, 16}

# From strings
text = "hello world"
unique_chars = {char for char in text if char != ' '}
print(unique_chars)  # {'h', 'e', 'l', 'o', 'w', 'r', 'd'}

Set Comprehensions with Conditions:

python

# Filter elements
numbers = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
even_squares = {x ** 2 for x in numbers if x % 2 == 0}
print(even_squares)  # {64, 4, 36, 100, 16}

# Complex conditions
words = ["apple", "banana", "cherry", "date", "elderberry"]
long_vowel_words = {word for word in words if len(word) > 5 and any(vowel in word for vowel in 'aeiou')}
print(long_vowel_words)  # {'banana', 'elderberry'}

4. Generator Expressions

Purpose: Create generators that produce values on-the-fly (memory efficient).

Basic Syntax:

python

(expression for item in iterable)

Basic Examples:

python

# Generator expression (uses parentheses instead of brackets)
squares_gen = (x ** 2 for x in range(5))
print(squares_gen)  # <generator object <genexpr> at 0x...>

# Convert to list to see values
print(list(squares_gen))  # [0, 1, 4, 9, 16]

# Use directly in functions that take iterables
total = sum(x ** 2 for x in range(5))
print(total)  # 30

# Memory efficiency demonstration
import sys

# List comprehension - stores all values in memory
list_comp = [x ** 2 for x in range(1000000)]
print(f"List size: {sys.getsizeof(list_comp)} bytes")  # Large size

# Generator expression - generates values on demand
gen_exp = (x ** 2 for x in range(1000000))
print(f"Generator size: {sys.getsizeof(gen_exp)} bytes")  # Small size

Generator Expressions with Conditions:

python

# Filter with condition
even_squares_gen = (x ** 2 for x in range(10) if x % 2 == 0)

# Use the generator
for num in even_squares_gen:
    print(num, end=' ')  # 0 4 16 36 64
print()

# Chain generators
numbers = (x for x in range(10))
squares = (x ** 2 for x in numbers)
filtered = (x for x in squares if x % 2 == 0)

print(list(filtered))  # [0, 4, 16, 36, 64]

5. Custom Generators with yield

Purpose: Create more complex generators using functions.

Basic Generator Function:

python

def countdown(n):
    """Generator that counts down from n to 1."""
    while n > 0:
        yield n
        n -= 1

# Use the generator
for number in countdown(5):
    print(number, end=' ')  # 5 4 3 2 1
print()

# Manual iteration
counter = countdown(3)
print(next(counter))  # 3
print(next(counter))  # 2
print(next(counter))  # 1
# print(next(counter))  # StopIteration error

More Complex Generators:

python

def fibonacci(limit):
    """Generate Fibonacci numbers up to limit."""
    a, b = 0, 1
    while a <= limit:
        yield a
        a, b = b, a + b

# Use Fibonacci generator
print("Fibonacci numbers:")
for num in fibonacci(20):
    print(num, end=' ')  # 0 1 1 2 3 5 8 13
print()

def read_large_file(filename):
    """Generator to read large files line by line (memory efficient)."""
    with open(filename, 'r') as file:
        for line in file:
            yield line.strip()

# Simulate usage
# for line in read_large_file("huge_file.txt"):
#     process(line)

6. Iterators: iter() and next()

Purpose: Manual control over iteration.

Basic Usage:

python

# Create an iterator from a list
numbers = [1, 2, 3, 4, 5]
iterator = iter(numbers)

print(next(iterator))  # 1
print(next(iterator))  # 2
print(next(iterator))  # 3
print(next(iterator))  # 4
print(next(iterator))  # 5
# print(next(iterator))  # StopIteration error

Handling StopIteration:

python

numbers = [1, 2, 3]
iterator = iter(numbers)

while True:
    try:
        item = next(iterator)
        print(item)
    except StopIteration:
        print("No more items")
        break

Custom Iterator Class:

python

class CountUpTo:
    """Custom iterator that counts up to a specified number."""
    
    def __init__(self, max_value):
        self.max_value = max_value
        self.current = 0
    
    def __iter__(self):
        return self
    
    def __next__(self):
        if self.current >= self.max_value:
            raise StopIteration
        self.current += 1
        return self.current

# Use custom iterator
counter = CountUpTo(5)
for num in counter:
    print(num, end=' ')  # 1 2 3 4 5
print()

# Manual iteration
counter = CountUpTo(3)
print(next(counter))  # 1
print(next(counter))  # 2
print(next(counter))  # 3
# print(next(counter))  # StopIteration

7. Practical Examples

Example 1: Data Processing Pipeline

python

def data_processing_pipeline(data):
    """Complete data processing using comprehensions and generators."""
    
    # Step 1: Clean data (remove None and empty strings)
    cleaned_data = [item.strip() for item in data if item and item.strip()]
    
    # Step 2: Convert to lowercase and filter by length
    filtered_data = [item.lower() for item in cleaned_data if len(item) >= 3]
    
    # Step 3: Create frequency dictionary
    frequency = {item: filtered_data.count(item) for item in set(filtered_data)}
    
    # Step 4: Generator for items with frequency > 1
    duplicates = (item for item, count in frequency.items() if count > 1)
    
    return {
        'cleaned_data': cleaned_data,
        'filtered_data': filtered_data,
        'frequency': frequency,
        'duplicates': list(duplicates)
    }

# Test the pipeline
sample_data = [" apple ", "banana", "", None, "Apple", "cat", "banana", "dog"]
results = data_processing_pipeline(sample_data)

print("Cleaned data:", results['cleaned_data'])
print("Filtered data:", results['filtered_data'])
print("Frequency:", results['frequency'])
print("Duplicates:", results['duplicates'])

Example 2: Configuration Parser

python

def parse_config_lines(lines):
    """Parse configuration lines using comprehensions."""
    
    # Remove comments and empty lines, strip whitespace
    config_lines = [line.strip() for line in lines 
                   if line.strip() and not line.strip().startswith('#')]
    
    # Parse key=value pairs
    config_dict = {}
    for line in config_lines:
        if '=' in line:
            key, value = line.split('=', 1)
            config_dict[key.strip()] = value.strip()
    
    # Create a generator for specific types
    def get_values_of_type(value_type):
        return (f"{k}: {v}" for k, v in config_dict.items() 
                if isinstance(v, value_type))
    
    return {
        'config': config_dict,
        'string_values': list(get_values_of_type(str)),
        'all_keys': [k for k in config_dict.keys()]
    }

# Sample configuration lines
config_lines = [
    "# Database configuration",
    "host = localhost",
    "port = 5432",
    "database = myapp",
    "",
    "# App settings",
    "debug = true",
    "max_connections = 100"
]

parsed = parse_config_lines(config_lines)
print("Configuration:", parsed['config'])
print("String values:", parsed['string_values'])
print("All keys:", parsed['all_keys'])

Example 3: Matrix Operations

python

def matrix_operations():
    """Various matrix operations using comprehensions."""
    
    # Create matrices
    matrix_a = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
    matrix_b = [[9, 8, 7], [6, 5, 4], [3, 2, 1]]
    
    # Transpose matrix
    transpose_a = [[row[i] for row in matrix_a] for i in range(len(matrix_a[0]))]
    print("Transpose of A:")
    for row in transpose_a:
        print(row)
    
    # Matrix addition
    matrix_sum = [[matrix_a[i][j] + matrix_b[i][j] for j in range(len(matrix_a[0]))] 
                 for i in range(len(matrix_a))]
    print("\nMatrix A + B:")
    for row in matrix_sum:
        print(row)
    
    # Matrix multiplication
    result = [[sum(a * b for a, b in zip(row_a, col_b)) 
              for col_b in zip(*matrix_b)] 
             for row_a in matrix_a]
    print("\nMatrix A × B:")
    for row in result:
        print(row)
    
    # Flatten matrix and get unique values
    all_values = {num for row in matrix_a for num in row}
    print(f"\nUnique values in A: {sorted(all_values)}")

matrix_operations()

Example 4: Prime Number Generator

python

def is_prime(n):
    """Check if a number is prime."""
    if n < 2:
        return False
    for i in range(2, int(n ** 0.5) + 1):
        if n % i == 0:
            return False
    return True

def prime_generators():
    """Different ways to generate prime numbers."""
    
    # List comprehension (eager evaluation)
    primes_up_to_50 = [x for x in range(2, 51) if is_prime(x)]
    print(f"Primes up to 50: {primes_up_to_50}")
    
    # Generator expression (lazy evaluation)
    prime_gen = (x for x in range(2, 100) if is_prime(x))
    print("First 10 primes from generator:", [next(prime_gen) for _ in range(10)])
    
    # Custom generator function
    def primes():
        """Infinite prime number generator."""
        n = 2
        while True:
            if is_prime(n):
                yield n
            n += 1
    
    # Use the infinite generator
    prime_iterator = primes()
    first_5_primes = [next(prime_iterator) for _ in range(5)]
    print(f"First 5 primes: {first_5_primes}")

prime_generators()

Key Takeaways:

1. List Comprehensions: [x**2 for x in range(5)] - Fast, readable list creation

2. Dict Comprehensions: {k: v**2 for k, v in dict.items()} - Concise dictionary creation

3. Set Comprehensions: {x for x in list} - Automatic duplicate removal

4. Generator Expressions: (x**2 for x in range(5)) - Memory-efficient lazy evaluation

5. Generator Functions: Use yield for complex generation logic

6. Iterators: Use iter() and next() for manual iteration control

When to Use Which:

• List/Dict/Set Comprehensions: When you need all results immediately

• Generator Expressions: When working with large data or infinite sequences

• Custom Generators: When generation logic is complex

• Iterators: When you need fine-grained control over iteration

Use generator expressions for large datasets to save memory, and comprehensions for smaller datasets where you need all results at once.

Mastering comprehensions and iterators will make your code more Pythonic, efficient, and readable!