Python Regular Expressions (Regex)

1. Introduction to Regex

Purpose: Define search patterns using special syntax to match, extract, or replace text.

Basic Regex Components:

• Literals: Regular characters that match themselves (a, 1, @)

• Metacharacters: Special characters with meaning (., *, +, ?, ^, $, etc.)

• Character Classes: Sets of characters ([abc], [0-9], [a-zA-Z])

• Quantifiers: Specify how many times something should occur (*, +, ?, {n})

2. Python's re Module

python

import re

Basic Functions:

• re.search() - Find first match anywhere in string

• re.match() - Find match only at beginning of string

• re.findall() - Find all matches

• re.finditer() - Find all matches (returns iterator)

• re.sub() - Replace matches

• re.split() - Split string by pattern

3. Basic Pattern Matching

Literal Matching:

python

import re

text = "Hello, my name is Alice. Hello everyone!"

# Search for exact string
match = re.search("Hello", text)
if match:
    print(f"Found at position {match.start()} to {match.end()}")  # Found at position 0 to 5

# Find all occurrences
all_hellos = re.findall("Hello", text)
print(all_hellos)  # ['Hello', 'Hello']

Special Characters (Metacharacters):

python

text = "cat bat rat mat hat"

# . (dot) matches any single character except newline
matches = re.findall(".at", text)
print(matches)  # ['cat', 'bat', 'rat', 'mat', 'hat']

# ^ matches start of string
start_match = re.findall("^cat", text)
print(start_match)  # ['cat']

# $ matches end of string
end_match = re.findall("hat$", text)
print(end_match)  # ['hat']

4. Character Classes

Basic Character Classes:

python

text = "abc123 XYZ!@#"

# [abc] - matches any of a, b, or c
print(re.findall("[abc]", text))  # ['a', 'b', 'c']

# [a-z] - matches any lowercase letter
print(re.findall("[a-z]", text))  # ['a', 'b', 'c']

# [0-9] - matches any digit
print(re.findall("[0-9]", text))  # ['1', '2', '3']

# [A-Z] - matches any uppercase letter
print(re.findall("[A-Z]", text))  # ['X', 'Y', 'Z']

# [a-zA-Z0-9] - alphanumeric characters
print(re.findall("[a-zA-Z0-9]", text))  # ['a', 'b', 'c', '1', '2', '3', 'X', 'Y', 'Z']

Predefined Character Classes:

python

text = "Hello World 123! \n\t"

# \d - digits (same as [0-9])
print(re.findall(r"\d", text))  # ['1', '2', '3']

# \D - non-digits
print(re.findall(r"\D", text))  # All non-digit characters

# \w - word characters (letters, digits, underscore)
print(re.findall(r"\w", text))  # ['H', 'e', 'l', 'l', 'o', 'W', 'o', 'r', 'l', 'd', '1', '2', '3']

# \W - non-word characters
print(re.findall(r"\W", text))  # [' ', ' ', '!', ' ', '\n', '\t']

# \s - whitespace characters
print(re.findall(r"\s", text))  # [' ', ' ', ' ', '\n', '\t']

# \S - non-whitespace characters
print(re.findall(r"\S", text))  # All non-whitespace characters

Note: We use raw strings (r"pattern") to avoid having to escape backslashes.

Negated Character Classes:

python

text = "abc123!@#"

# [^abc] - matches anything EXCEPT a, b, or c
print(re.findall("[^abc]", text))  # ['1', '2', '3', '!', '@', '#']

# [^0-9] - matches non-digits
print(re.findall("[^0-9]", text))  # ['a', 'b', 'c', '!', '@', '#']

5. Quantifiers

Basic Quantifiers:

python

text = "a ab abb abbb abbbb"

# * - 0 or more occurrences
print(re.findall("ab*", text))  # ['a', 'ab', 'abb', 'abbb', 'abbbb']

# + - 1 or more occurrences
print(re.findall("ab+", text))  # ['ab', 'abb', 'abbb', 'abbbb']

# ? - 0 or 1 occurrence
print(re.findall("ab?", text))  # ['a', 'ab', 'ab', 'ab', 'ab']

# {n} - exactly n occurrences
print(re.findall("ab{2}", text))  # ['abb', 'abb'] (from 'abb' and 'abbb')

# {n,} - n or more occurrences
print(re.findall("ab{2,}", text))  # ['abb', 'abbb', 'abbbb']

# {n,m} - between n and m occurrences
print(re.findall("ab{2,3}", text))  # ['abb', 'abbb']

Greedy vs Non-Greedy Matching:

python

x

text = "a ab abb abbb abbbb"

# * - 0 or more occurrences
print(re.findall("ab*", text))  # ['a', 'ab', 'abb', 'abbb', 'abbbb']

# + - 1 or more occurrences
print(re.findall("ab+", text))  # ['ab', 'abb', 'abbb', 'abbbb']

# ? - 0 or 1 occurrence
print(re.findall("ab?", text))  # ['a', 'ab', 'ab', 'ab', 'ab']

# {n} - exactly n occurrences
print(re.findall("ab{2}", text))  # ['abb', 'abb'] (from 'abb' and 'abbb')

# {n,} - n or more occurrences
print(re.findall("ab{2,}", text))  # ['abb', 'abbb', 'abbbb']

# {n,m} - between n and m occurrences
print(re.findall("ab{2,3}", text))  # ['abb', 'abbb']

6. Grouping and Capturing

Basic Groups:

python

text = "John: 30, Alice: 25, Bob: 35"

# Parentheses create capturing groups
pattern = r"(\w+): (\d+)"
matches = re.findall(pattern, text)
print(matches)  # [('John', '30'), ('Alice', '25'), ('Bob', '35')]

# Using finditer to get match objects
for match in re.finditer(pattern, text):
    print(f"Full match: {match.group(0)}")
    print(f"Name: {match.group(1)}, Age: {match.group(2)}")
    print(f"Position: {match.start()} to {match.end()}")

Named Groups:

python

text = "Date: 2024-12-25, Time: 14:30"

# Named groups with ?P<name>
pattern = r"Date: (?P<year>\d{4})-(?P<month>\d{2})-(?P<day>\d{2})"
match = re.search(pattern, text)

if match:
    print(f"Year: {match.group('year')}")   # 2024
    print(f"Month: {match.group('month')}") # 12
    print(f"Day: {match.group('day')}")     # 25
    print(f"All groups: {match.groupdict()}")  # {'year': '2024', 'month': '12', 'day': '25'}

Non-Capturing Groups:

python

text = "color colour"

# (?:) - group but don't capture
pattern = r"col(o|ou)r"  # Capturing group
print(re.findall(pattern, text))  # ['o', 'ou']

pattern = r"col(?:o|ou)r"  # Non-capturing group
print(re.findall(pattern, text))  # ['color', 'colour']

7. Alternation and Boundaries

Alternation (OR):

python

x

text = "color colour"

# (?:) - group but don't capture
pattern = r"col(o|ou)r"  # Capturing group
print(re.findall(pattern, text))  # ['o', 'ou']

pattern = r"col(?:o|ou)r"  # Non-capturing group
print(re.findall(pattern, text))  # ['color', 'colour']

Word Boundaries:

python

text = "cat category catfish concatenate"

# \b - word boundary
word_boundary = re.findall(r"\bcat\b", text)
print("Word boundary:", word_boundary)  # ['cat'] (only standalone 'cat')

# Without word boundary
no_boundary = re.findall(r"cat", text)
print("No boundary:", no_boundary)  # ['cat', 'cat', 'cat', 'cat'] (all occurrences)

8. Search and Match Methods

re.search() vs re.match():

python

text = "Hello World! Hello Python!"

# search() finds first match anywhere in string
search_result = re.search("World", text)
if search_result:
    print(f"search found: '{search_result.group()}' at position {search_result.start()}")

# match() only looks at beginning of string
match_result = re.match("World", text)
print("match found:", match_result)  # None

match_result = re.match("Hello", text)
if match_result:
    print(f"match found: '{match_result.group()}'")  # 'Hello'

re.findall() vs re.finditer():

python

text = "cat bat rat mat"

# findall() returns list of strings
matches_list = re.findall(r".at", text)
print("findall:", matches_list)  # ['cat', 'bat', 'rat', 'mat']

# finditer() returns iterator of match objects
matches_iter = re.finditer(r".at", text)
for match in matches_iter:
    print(f"Found '{match.group()}' at position {match.start()}-{match.end()}")

9. Substitution with re.sub()

Basic Substitution:

python

text = "Hello World! Hello Python!"

# Simple replacement
new_text = re.sub(r"Hello", "Hi", text)
print(new_text)  # "Hi World! Hi Python!"

# Using captured groups in replacement
text = "John Smith"
new_text = re.sub(r"(\w+) (\w+)", r"\2, \1", text)  # Swap first and last name
print(new_text)  # "Smith, John"

Advanced Substitution with Functions:

python

def double_numbers(match):
    """Double the matched number."""
    number = int(match.group())
    return str(number * 2)

text = "I have 5 apples and 3 oranges"
new_text = re.sub(r"\d+", double_numbers, text)
print(new_text)  # "I have 10 apples and 6 oranges"

# Using lambda functions
text = "CAT dog BIRD"
new_text = re.sub(r"[A-Z]+", lambda m: m.group().lower(), text)
print(new_text)  # "cat dog bird"

10. Splitting with re.split()

python

text = "apple,banana;cherry:date.elderberry"

# Split by multiple delimiters
parts = re.split(r"[,;:.]", text)
print(parts)  # ['apple', 'banana', 'cherry', 'date', 'elderberry']

# Split with capture groups
text = "word1, word2, word3"
parts = re.split(r"(\W+)", text)  # Capture delimiters too
print(parts)  # ['word1', ', ', 'word2', ', ', 'word3']

# Split with maxsplit
text = "1-2-3-4-5"
parts = re.split(r"-", text, maxsplit=2)  # Only split first 2 times
print(parts)  # ['1', '2', '3-4-5']

11. Compiling Regex Patterns

Purpose: Pre-compile patterns for better performance when reusing the same pattern.

python

# Compile pattern once, use multiple times
pattern = re.compile(r"\b\w{4}\b")  # 4-letter words

text1 = "This is a test sentence with some words"
text2 = "Another example with different words"

matches1 = pattern.findall(text1)
matches2 = pattern.findall(text2)

print("Text1 matches:", matches1)  # ['This', 'test', 'some']
print("Text2 matches:", matches2)  # ['with']

# All re functions are available as methods
compiled_pattern = re.compile(r"(\d{3})-(\d{3})-(\d{4})")  # Phone number pattern

text = "Call me at 123-456-7890 or 987-654-3210"
matches = compiled_pattern.findall(text)
print("Phone numbers:", matches)  # [('123', '456', '7890'), ('987', '654', '3210')]

# Substitution with compiled pattern
new_text = compiled_pattern.sub(r"(\1) \2-\3", text)
print("Formatted:", new_text)  # "Call me at (123) 456-7890 or (987) 654-3210"

12. Practical Examples

Example 1: Email Validation

python

def validate_email(email):
    """Validate email address format."""
    pattern = r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'
    return bool(re.match(pattern, email))

# Test emails
emails = [
    "user@example.com",
    "hello.world@company.co.uk", 
    "invalid.email",
    "missing@tld.",
    "user@sub.domain.com"
]

for email in emails:
    is_valid = validate_email(email)
    print(f"{email:30} -> {'Valid' if is_valid else 'Invalid'}")

Example 2: Phone Number Extractor

python

def extract_phone_numbers(text):
    """Extract and format phone numbers from text."""
    # Multiple phone number formats
    patterns = [
        r'\b\d{3}-\d{3}-\d{4}\b',          # 123-456-7890
        r'\b\(\d{3}\) \d{3}-\d{4}\b',      # (123) 456-7890
        r'\b\d{3}\.\d{3}\.\d{4}\b',        # 123.456.7890
        r'\b\d{10}\b',                     # 1234567890
        r'\b\d{3} \d{3} \d{4}\b'           # 123 456 7890
    ]
    
    phone_numbers = []
    
    for pattern in patterns:
        matches = re.findall(pattern, text)
        for match in matches:
            # Standardize format to (123) 456-7890
            digits = re.sub(r'\D', '', match)  # Remove non-digits
            if len(digits) == 10:
                formatted = f"({digits[:3]}) {digits[3:6]}-{digits[6:]}"
                phone_numbers.append(formatted)
    
    return list(set(phone_numbers))  # Remove duplicates

# Test text with various phone formats
text = """
Contact us at 123-456-7890 or (987) 654-3210. 
You can also reach us at 555.123.4567, 8887654321, 
or at 111 222 3333. Don't call 123-45-6789 (invalid).
"""

numbers = extract_phone_numbers(text)
print("Found phone numbers:")
for number in numbers:
    print(f"  - {number}")

Example 3: Log File Parser

python

def parse_log_file(log_text):
    """Parse log file entries and extract structured data."""
    
    # Common log format: [TIMESTAMP] LEVEL: Message (User: username)
    pattern = r'\[(?P<timestamp>[^\]]+)\] (?P<level>\w+): (?P<message>[^\(]+)(?:\(User: (?P<user>\w+)\))?'
    
    log_entries = []
    
    for match in re.finditer(pattern, log_text):
        entry = match.groupdict()
        # Clean up the message
        entry['message'] = entry['message'].strip()
        # If no user, set to 'system'
        entry['user'] = entry['user'] or 'system'
        log_entries.append(entry)
    
    return log_entries

# Sample log data
log_data = """
[2024-01-15 10:30:25] INFO: User logged in successfully (User: alice)
[2024-01-15 10:31:10] ERROR: Database connection failed (User: bob)
[2024-01-15 10:32:45] WARNING: High memory usage detected
[2024-01-15 10:33:20] INFO: File uploaded successfully (User: charlie)
"""

entries = parse_log_file(log_data)
print("Parsed log entries:")
for i, entry in enumerate(entries, 1):
    print(f"{i}. [{entry['timestamp']}] {entry['level']}: {entry['message']} (User: {entry['user']})")

Example 4: HTML/XML Tag Extractor

python

def extract_tags(html_text):
    """Extract all HTML/XML tags and their content."""
    
    # Pattern to match tags and their content
    pattern = r'<(\w+)(?:\s+[^>]*)?>([^<]*)</\1>'
    
    tags = []
    
    for match in re.finditer(pattern, html_text):
        tag_name = match.group(1)
        content = match.group(2).strip()
        tags.append({
            'tag': tag_name,
            'content': content,
            'full_match': match.group(0)
        })
    
    return tags

# Sample HTML
html_content = """
<div class="header">Welcome</div>
<p>This is a paragraph with <strong>bold text</strong> inside.</p>
<span id="main">Main content here</span>
<a href="https://example.com">Click here</a>
"""

tags = extract_tags(html_content)
print("Extracted tags:")
for tag_info in tags:
    print(f"<{tag_info['tag']}>: '{tag_info['content']}'")

Example 5: Password Strength Checker

python

def check_password_strength(password):
    """Check password strength using regex patterns."""
    
    checks = {
        'length': len(password) >= 8,
        'lowercase': bool(re.search(r'[a-z]', password)),
        'uppercase': bool(re.search(r'[A-Z]', password)),
        'digit': bool(re.search(r'\d', password)),
        'special_char': bool(re.search(r'[!@#$%^&*(),.?":{}|<>]', password)),
        'no_spaces': not re.search(r'\s', password),
        'no_common_patterns': not re.search(r'(123|abc|password|qwerty)', password, re.IGNORECASE)
    }
    
    score = sum(checks.values())
    feedback = []
    
    if not checks['length']:
        feedback.append("Must be at least 8 characters long")
    if not checks['lowercase']:
        feedback.append("Must contain lowercase letters")
    if not checks['uppercase']:
        feedback.append("Must contain uppercase letters")
    if not checks['digit']:
        feedback.append("Must contain numbers")
    if not checks['special_char']:
        feedback.append("Must contain special characters")
    if not checks['no_spaces']:
        feedback.append("Must not contain spaces")
    if not checks['no_common_patterns']:
        feedback.append("Avoid common patterns")
    
    strength_levels = {
        7: "Very Strong",
        6: "Strong",
        5: "Good",
        4: "Fair", 
        3: "Weak",
        2: "Very Weak"
    }
    
    return {
        'score': score,
        'strength': strength_levels.get(score, "Very Weak"),
        'feedback': feedback,
        'checks': checks
    }

# Test passwords
passwords = [
    "weak",
    "Better123",
    "Strong@Pass123!",
    "12345678",
    "NoSpecialChar123"
]

for pwd in passwords:
    result = check_password_strength(pwd)
    print(f"Password: {pwd}")
    print(f"Strength: {result['strength']} (Score: {result['score']}/7)")
    for feedback in result['feedback']:
        print(f"  {feedback}")
    print()

13. Common Regex Patterns

Useful Patterns:

python

# Common patterns dictionary
PATTERNS = {
    'email': r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$',
    'phone_us': r'^\(?\d{3}\)?[-.\s]?\d{3}[-.\s]?\d{4}$',
    'url': r'https?://(?:[-\w.]|(?:%[\da-fA-F]{2}))+',
    'ip_address': r'\b(?:\d{1,3}\.){3}\d{1,3}\b',
    'credit_card': r'\b\d{4}[- ]?\d{4}[- ]?\d{4}[- ]?\d{4}\b',
    'zip_code': r'\b\d{5}(?:-\d{4})?\b',
    'date_ymd': r'\b\d{4}-\d{2}-\d{2}\b',
    'time_24h': r'\b([01]?[0-9]|2[0-3]):[0-5][0-9]\b',
    'hex_color': r'#([a-fA-F0-9]{6}|[a-fA-F0-9]{3})\b',
    'html_tag': r'</?(\w+)(?:\s+[^>]*)?>'
}

def quick_match(pattern_name, text):
    """Quickly test common patterns."""
    if pattern_name in PATTERNS:
        return bool(re.search(PATTERNS[pattern_name], text))
    return False

# Test the patterns
test_cases = [
    ("email", "user@example.com"),
    ("phone_us", "123-456-7890"),
    ("url", "https://example.com"),
    ("ip_address", "192.168.1.1")
]

for pattern_name, test_text in test_cases:
    matches = quick_match(pattern_name, test_text)
    print(f"{pattern_name:12} '{test_text}' -> {'Match' if matches else 'No match'}")

Key Takeaways:

1. re.search() - Find first match anywhere

2. re.match() - Find match only at beginning

3. re.findall() - Find all matches as list

4. re.finditer() - Find all matches as iterator

5. re.sub() - Replace matches

6. re.split() - Split by pattern

7. re.compile() - Pre-compile for performance

Common Metacharacters:

• . Any character except newline

• * 0 or more, + 1 or more, ? 0 or 1

• ^ Start of string, $ End of string

• \b Word boundary, \B Not word boundary

• \d Digit, \D Not digit, \w Word char, \W Not word char

• [] Character class, [^] Negated class

• | Alternation (OR)

• () Capturing group, (?:) Non-capturing group