Professional Documents
Culture Documents
Jimmy Guerrero
Senior Product Manager
jguerrero@mysql.com
Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 1
Agenda
• Introduction to MySQL
• Comparing MySQL HA Architectures
– MySQL Replication
– MySQL Replication plus Linux Heartbeat
– MySQL Cluster
• MySQL, Heartbeat plus DRBD Architectures
• Setup and Configuration
• Managing Failover
• High Availability Services from MySQL
• Q&A
Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 2
Overview of MySQL AB
• Founded in 1995
• Operations in 24 countries
• World’s Most Popular Open Source Database
– Over 10 million installations
– 50,000 downloads each day
• Reduces TCO of database management
– Reliable, high performance, easy to use
• Bundled by over 100 software & hardware companies
• Partners
HP, Dell, Novell, Red Hat, Unisys, Accenture, LogicaCMG, T-Systems and IPL
Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 3
Established & Emerging Companies
High Volume Websites
– Web 2.0
– Dynamic content
– eCommerce
– Look to Book
– Session Management
– Gaming & Entertainment craigslist
– Scale Out
Enterprise
– Data Warehousing
– High-Volume OLTP
– Scale Out
Embedded
– Software & Hardware
Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 4
MySQL, Heartbeat and DRBD Cluster
• High availability solution built on open source software
– MySQL: Database
– Linux Heartbeat: Virtual IP management
– DRBD: Synchronous block level replication and fast data
resynchronization after node failure
• Low cost
– Many popular Linux distributions supported
– Can be built on COTS hardware components
– No migration hassles
• Supports MySQL “scale out” for reads
• Supports transactional storage engines
– InnoDB
– Falcon
Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 5
What is DRBD?
• Block device which leverages synchronous replication
– Open source Linux kernel module
– Bundled by several popular Linux distributions
– Project maintained by LinBit
• Core functionality
– The entire block device is mirrored via a dedicated network
– Similar to RAID-1 over Ethernet
– DRBD operates at disk I/O layer
– Transparent to MySQL and the underlying file system
– MySQL fails over to passive DRBD server via "virtual IP
management"
• Typically accomplished with the Linux Heartbeat package.
– MySQL transactional storage engine independent
• InnoDB, Falcon
http://www.drbd.org/ http://www.linux-ha.org/
Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 6
MySQL HA Solutions
Clustering &
Geographical
Redundancy
MySQL
Clustering Cluster
Technologies
MySQL + DRBD
Replication
MySQL + SAN
Technologies
Well-Managed MySQL
Replication
Unmanaged
9
35 days
9.
4 days
9
8 hours
9
50 mins
9
5 mins
%
Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 7
What is Replication?
Asynchronous Replication
update
customer
Synchronous Replication
update
customer
Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 8
MySQL Replication
• Asynchronous Requirements Support
– Statement Availability
– Row-based (5.1) Automated IP Fail Over No
– Mixed (5.1)
Automated DB Fail Over No
• Read vs. Write
Typical Fail Over Time Varies
– Response times
Resynchronization of Data No
– Load balancing
– Scale Out Geographic Redundancy Yes
• Administration Scalability
Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 9
MySQL Replication Topologies
Master > Slave Master > Slaves
Circular (Multi-Master)
Master < > Master (Multi-Master)
Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 10
MySQL Replication - Scale Out
• Write to one master
• Read from many slaves
• Perfect for read intensive apps
Load Balancer
MySQL Replication
Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 11
MySQL Replication & Linux Heartbeat
IP Management
Active Passive
MySQL MySQL
Server Server
Asynchronous
MySQL Replication
Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 12
MySQL Cluster
Requirements Support
• In-Memory Database
Availability
– Disk Data (5.1)
• Shared-nothing storage Automated IP Fail Over No
• Fast-automatic data fail over Typical Fail Over Time < 3 Secs
5.0: 64
# of Nodes
5.1: 255
Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 13
MySQL Cluster: Architecture
Clients
Management
Nodes
(Typically 2)
Data Nodes
Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 14
MySQL Cluster: Data Distribution
Two Replicas
* Two copies of each partition
Node Group
Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 15
MySQL, Heartbeat & DRBD Cluster
• Distributed storage Requirements Support
• Similar to network RAID 1 Availability
# of Nodes 1 Active
Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 16
MySQL, Heartbeat & DRBD Cluster
IP Management
Active Passive
DRBD DRBD
Server Server
Synchronous
Block Replication
Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 17
Scale Out
MySQL Replication w/ DRBD Cluster
IP Management
Active Passive
Server Server
Shard 1 Shard 2
IP Management IP Management
Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 20
Block Devices
• A device node is a type of special file
– Used on many Unix-like operating systems
• Facilitates transparent communication
– Between user space applications and hardware
• Correspond to resources allocated by the operating
system kernel
• Accessed using standard system calls and treated like
regular files
• Used to correspond to devices
– Data is transmitted in the form of blocks.
• Make use of buffered input and output routines
Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 21
DRBD Basics
• Each device has one of two states
– Primary
– Secondary
• Application runs and accesses the device on the
primary node
• Writes are sent to the local lower level block device
and to the node with the device in secondary state
• Secondary device writes the data to its lower level
block device
• Reads are always carried out locally
– Key advantage over SAN/NAS
• If the primary node fails
– Heartbeat switches the secondary device into a primary state
and starts the application (MySQL)
Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 22
DRBD Basics
• When the failed node comes up
– Becomes a secondary node
– Must synchronize its data to the primary
– Synching happens in the background without service
interruption
– Only the parts of the device which have changed since the
failure will be resynchronized.
• Total resynchronization in just several minutes
– Regardless of device size
– Even after the hard crash of an active node
– Resynch rates may be configured per device
• Device sizes currently limited to 4 TB
Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 23
Setup & Configuration
Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 24
Setup & Configuration
• Multiple NICs will increase redundancy
• MySQL traffic switched over public LAN
• DRBD & Heartbeat over private LAN
IP Management
Active Passive
DRBD DRBD
Server Server
Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 25
Setup & Configuration - DRBD
• Additional prerequisite packages may be needed
– Depending on kernel version
– When building from source
– Or using RPMs
• Pre-built x86 and x86_64 packages for specific kernel
versions available with a support subscription
– SuSE 9 & 10
– Redhat AS 4
• If kernel is upgraded, DRBD must be as well
• Current DRBD version is 8 (Jan 24, 2007)
– Better, tuneable recovery
– Ability to run DRBD on top of LVM
– Performance optimization (Jumbo frames)
Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 26
Setup & Configuration - DRBD
• Install DRBD package
• Edit and copy configuration file (drbd.conf) to nodes
• Choose primary node
• Synchronize the underlying devices
• Device is now ready
– Create a file system if one does not exist
• Mount DRBD on primary
• Heartbeat handles
– Changing of DRBD primary or secondary status
– Mounting and unmounting volumes
Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 27
Setup & Configuration - MySQL
• Ensure all MySQL files are installed on the DRBD
volume
• Create ‘mysql’ group and user
• Create MySQL directory
• Install MySQL
• Shutdown MySQL
• Unmount the DRBD volume
Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 28
Setup & Configuration - Heartbeat
• Install Heartbeat V2 package
– http://www.linux-ha.org/
– Ships with many Linux distributions
• Configuration
– ha.cf: describes the participating machines and their
configuration
– haresources: specifies virtual IP and managed services
– ipfiles: detects when one of their network links has become
unusable, and compensate accordingly
– authkeys: three types - crc, md5, and sha1
– resource.d/mysql: start and stop script for MySQL
Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 29
Initialization & Next Steps
• Start DRBD
• Start Heartbeat
– Node becomes DRBD primary
– DRBD volume is mounted
– MySQL starts
– IP address taken over
• Next steps
– Fail over testing
– Benchmarking
– Stress testing
– Create monitoring framework
– Attach MySQL Replication slaves for read scalability
Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 30
Managing Failures
IP Management
Passive X Active
DRBD DRBD
Server Server
X
X
X
Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 31
Data Resynchronization after Host Failure
IP Management
Passive Active
DRBD DRBD
Server Server
Resynchronization
Typical Fail Over Time Varies Varies < 30 secs < 3 secs
Scalability
MySQL
Built-in Load Balancing MySQL Replication MySQL Replication Yes
Replication
Dozens for
# of Slaves Dozens for Reads Dozens for Reads Dozens for Reads
Reads
Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 33
Services for DRBD from MySQL
http://www.mysql.com/consulting/packaged/scaleout.html
Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 34
Case Study: footnote.com
• Online repository of historical documents & images
• Founded in 1994, based in Utah, 80 employees
Component Detail
Operating
CentOS 4.4
System
DB Size: 15 GB
Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 35
Summary
• High availability solution built on open source software
– MySQL: Database
– Linux Heartbeat: Virtual IP management
– DRBD: Synchronous block level replication and fast data
resynchronization after node failure
• Low cost
– Many popular Linux distributions supported
– Can be built on COTS hardware components
– No migration hassles
• Supports MySQL “scale out” for reads
• Supports transactional storage engines
– InnoDB
– Falcon
Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 36
MySQL Conference & Expo 2007
April 23-27, Santa Clara
MySQL 5.1 – Falcon – Partitioning
Administration - DRBD
Thank You!
Jimmy Guerrero
Senior Product Manager
jimmy@mysql.com
White Papers - Case Studies - Press Releases - News - Events - Live Webinars - Forums
Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 38