Percona Live Amsterdam 2016 Notes

Early this month I attended Percona Live, below are some random notes:

MySQL 8.0

MySQL 8.0 is currently quite far out, with a DMR available. While there is no official release date yet, another 2 years or so is likely.

Some companies are testing the DMR in production to weed out potential issues during the development.

Some exciting features include

InnoDB by default!

MySQL 5.7 is still using MyISAM for system tables, 8.0 will see MyISAM still shipped, but completely optional due to the new data dictionary.

Another feature request has been raised to move the storage engine to a plugin, with some comments suggesting a separate package.

All the features in MyISAM have been implemented in InnoDB (5.6, 5.7), with MyISAM still performing better in some cases.

Some key reasons why not to use MyISAM unless there's a specific use case (also applies to 5.6/5.7):

  • Not ACID compliant
  • Not transactional
  • Repair + index rebuild required on crashes
  • Table level locking

Invisible indexes

A very interesting feature, allowing you to hide an index from the query optimizer, but critically keep the index updated.

There is a feature in MyISAM called 'disabled indexes', which has differing behaviour; this stops the index being maintained, requiring a rebuild of the index when it's re-enabled.

Why is this useful?

  • Dropping an index; once dropped rebuilding the index could be very time-consuming and in a production setting, this could have a significant effect on your applications. Hiding the index would reveal the same performance hit, with a low-cost rollback.

  • Adding an index; creating a new index can cause the query plan to change, sometimes negatively. Currently, it's not possible to 'force' a hidden index to be on, this would be an interesting feature; enabling testing of critical queries.

IPv6 comparisons

INET6_ATON / INET6_NTOA were added in MySQL 5.6 however binary comparison has never worked properly. WL#8699 has changed that.

You can now do INET6_ATON(address) & INET6_ATON(network) and the result is correct.

A small change, but for certain applications, a lot of logic around v6 can now be treated the same way as v4.

JSON support

The JSON support continues to mature, with 2 new functions added in 8.0; JSON_ARRAYAGG & JSON_OBJECTAGG.

These functions work on tables without JSON fields, allowing logic such as:

mysql> SELECT JSON_ARRAYAGG(`name`) AS `devices` FROM `assets` where `active` = b'1';
+--------------------------+
| devices                  |
+--------------------------+
| [ "device1",
    "device2",
    "device3" ]
|
+--------------------------+

Or

mysql> SELECT JSON_OBJECTAGG(`name`, `ip`) AS `devices` FROM `assets` where `active` = b'1';
+--------------------------+
| devices                  |
+--------------------------+
| { "device1": "127.0.0.1",
    "device2": "127.0.0.2",
    "device3": "127.0.0.3" }
|
+--------------------------+

Other SQL logic should as grouping or joins can also be performed

mysql> SELECT `location`, JSON_OBJECTAGG(`name`, `ip`) AS `devices` FROM `assets` where `active` = b'1' group by `location`;
+--------------+--------------------------------------------------+
| location     | devices                                          |
+--------------+--------------------------------------------------+
| Amsterdam    | {"device1": "127.0.0.1", "device2": "127.0.0.2"} |
| Manchester   | {"device3": "127.0.0.3"}                         |
+--------------+--------------------------------------------------+

I'm not sure how much application/serialisation logic I want in my database, but this is definitely powerful.

CTE

This literally hurts my brain, seemingly it can be used in place of derived tables, with 1 powerful feature; you can recurse over the statement.

In the case of a table being referenced to its self, it's now simple (and in some cases possible) to query the data in 1 pass.

The CTE only knows about 1 row at a time, so for really complex items, columns have to be passed around (see this post for a Fibonacci example).

A simple example is as below:

WITH RECURSIVE crazy AS
(
  SELECT 1 AS counter
  UNION ALL
  SELECT 2 + counter FROM crazy WHERE counter < 10
)
SELECT * FROM crazy;

+---------+
| counter |
+---------+
|       1 |
|       3 |
|       5 |
|       7 |
|       9 |
+---------+

Other stuff

Lots of other nice features have been added such as:

  • Better docs
  • Better GIS support
  • Performance improvements
  • Improved security models
  • Query optimizer cost model improvements
  • Support for the latest Unicode 9.0 standard
  • UTF8MB4 as the default character set (work started in 5.7)

Follow the MySQL server team blog for updates.

ProxySQL

ProxySQL is a high-performance SQL proxy, supporting a number of interesting features:

  • Query rewriting / blocking
  • Load balancing / query routing
  • Caching
  • HA (when used with a topology manager)
  • Traffic mirroring

The HA part is especially interesting for a few reasons:

  • Applications have their connection held, so you can have 'no downtime, increased latency' master failovers
  • Servers can be promoted based on their read_only status
  • Integration with cluster managers (Galera etc) is possible

Marco Tusa had a presentation on deploying ProxySQL, with an interesting overview of failover techniques and what 'HA' means:

90.000% (36 days) MySQL Replication
99.900% (8 hours) Linux Heartbeat with DRBD
99.900% (8 hours) RHCS with Shared Storage (Active/Passive)
99.990% (52 minutes) MHA with at least 3 nodes
99.990% (52 minutes) Linux Heartbeat with DRBD and Replication
99.990% (52 minutes) Linux Heartbeat with Shared Storage and Replication
99.995% (26 minutes) Multi-Master (Galera - Percona cluster) 3 node minimum
99.999% (5 minutes) MySQL Cluster

gh-ost

Think pt-online-schema-change without triggers, on steroids.

The killer features include:

  • No triggers, so 0 performance overhead when paused
  • Dynamic throttling
  • Hooks for notifications / cut-over
  • Re-configurable at runtime
  • Pauseable

Currently, they're still using the INSERT INTO xxx SELECT FROM yyy pattern for the bulk row copy (binlog events played over the top), there is a feature request for this to be done via the gh-ost process; if that happens a number of really cool possibilities open up:

  • No extra read load on the masters (and no touching the source table) - read data from slaves
  • Live table migrations - no complex intermediary master setup for migrations/vertical sharding
  • Sharding - with support for conditionals, slicing a table in 2 could be done in the same way as a live table migration

The only catch is you need row based binlogs (binlog_row_image=FULL, though MINIMAL has been requested); this can be supported in a statement or mixed topology by having 1 slave log changes in the row format.

Facebook backups / binlog server

Facebook had a number of talks, 2 of which were on the topic of binlogs and backups.

Interestingly their strategy is along the lines of:

  • Backup binlogs as they rotate
  • Stream binlogs into HDFS
  • mysqldump all databases every day

The dumping process is interesting, as they take a full backup, but only save that every 5 days. The rest of the days, they compute the updated (+inserted) and deleted rows, then store that.

When restoring a database/table they:

  1. Download the last full backup
  2. Download the diff backups between full...now
  3. Combine the full + diffs into 1 file
  4. Restore that file
  5. Replay the binlogs

The only benefit I can see to this (over just storing the full backups) is saving on storage space (assuming < 100% churn of the data).

They have made multiple additions to the mysqldump binary (available in their mysql-5.6 branch), such as storing table offsets in comments.

For restores, there is an orchestrated pipeline with a scheduler submitting automated restore requests to test backups; this has picked up failures in the process previously.

To handle binlogs, they wrote their own binlog server, which can serve the files out of HDFS upon request; this is used to seed slaves and replay entries on restores.

MyRocks

It was announced that Percona Server will be supporting MyRocks, this is exciting as previously you needed to run Facebook's branch of MySQL.

I've not yet had time to play with MyRocks, but some key stats posted by others when compared to InnoDB:

  • 1.5x more queries per second
  • 8x-14x less data written per transaction
  • 2x-3x less space used

Follow Mark Callaghan for interesting RocksDB updates.

Percona Monitoring and Management

This is still pretty new, but there's a nice bundle of tools. I'm interested to see how this project matures and if the pain of MySQL Enterprise Monitor can be forgotten.

There's an online demo if you want to play with it.

Database engine licensing

Peter Zaitsev's keynote reminded me of the excellent db-engines.com and their trends on licensing and usage.

The popularity of Opensource engine's is growing with some of the biggest users of commercial licenses being their owners (Microsoft, Oracle).

It also highlighted that most commercial offerings are around relational stores, with Opensource heavily present in wide column, graph, key-value and time series stores.

A key takeaway for this segment was 'free' really means reduced TCO for business; they pay the support cost in people/consultancy etc but with a 3-10x reduction in TCO.

GNS3 crash after upgrade

Recently I upgraded the GNS3 version on my Macbook and afterwards, immediately after trying to open the app it would close.

Attempting to start the processes on the command line revealed some additional information:

  • /Applications/GNS3.app/Contents/MacOS/gns3server was working as expected
  • /Applications/GNS3.app/Contents/MacOS/gns3shell also was working as expected
  • /Applications/GNS3.app/Contents/MacOS/gns3 was crashing with ImportError: No module named 'gns3.main'
  • /Applications/GNS3.app/Contents/MacOS/Python was not executable

Having also recently updated homebrew and upon deciding to google the error, a good 30min was wasted tracking down possible python path issues.

As it turns out, GNS3 on OSX is built using cx_Freeze, ensuring there are minimal dependencies on the system.

So, what is the problem? To find that out, lets look at the full output:

m00m00:~ damian$ /Applications/GNS3.app/Contents/MacOS/gns3
Application frozen with cx_Freeze
GNS3 GUI version 1.5.2
Copyright (c) 2007-2016 GNS3 Technologies Inc.
INFO logger.py:107 Log level: INFO
INFO servers.py:689 New remote server connection http://192.168.95.128:8000 registered
INFO local_config.py:294 Section LocalServer has changed. Saving configuration
INFO local_config.py:190 Configuration save to /Users/damian/.config/GNS3/gns3_gui.conf
****** Exception detected, traceback information saved in /Users/damian/.config/GNS3/exceptions.log ******

PLEASE REPORT ON https://www.gns3.com

Traceback (most recent call last):
  File "/Users/gns3/Jenkins/workspace/Release MacOSX/OSX/../boot.py", line 35, in <module>
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/importlib/__init__.py", line 126, in import_module
  File "<frozen importlib._bootstrap>", line 986, in _gcd_import
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 956, in _find_and_load_unlocked
ImportError: No module named 'gns3.main'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/gns3/Jenkins/workspace/Release MacOSX/OSX/../boot.py", line 46, in <module>
  File "gns3-gui/gns3/main.py", line 281, in <module>
  File "gns3-gui/gns3/main.py", line 244, in main
  File "./gns3-gui/gns3/main_window.py", line 109, in __init__
  File "./gns3-gui/gns3/ui/main_window_ui.py", line 188, in setupUi
  File "./gns3-gui/gns3/server_summary_view.py", line 101, in __init__
  File "./gns3-gui/gns3/servers.py", line 875, in instance
  File "./gns3-gui/gns3/servers.py", line 74, in __init__
  File "./gns3-gui/gns3/servers.py", line 268, in _loadSettings
  File "./gns3-gui/gns3/servers.py", line 277, in _saveSettings
  File "./gns3-gui/gns3/servers.py", line 277, in <listcomp>
KeyError: 'url'

As it turns out, the issue appears to be in loading the config file.

There is no 'url' key within the config file I have on disk, so let's remove it and try again.

Same problem, strace...err dtruss to the rescue!

m00m00:~ damian$ sudo dtruss /Applications/GNS3.app/Contents/MacOS/gns3

Application frozen with cx_Freeze
GNS3 GUI version 1.5.2
Copyright (c) 2007-2016 GNS3 Technologies Inc.
[....]
stat64("/Users/damian/.config/GNS3/GNS3.ini\0", 0x7FFF57315A68, 0x0)                = 0 0
[....]

As some of you may have noticed in the stacktrace, the config that is being correctly saved is /Users/damian/.config/GNS3/gns3_gui.conf, yet we're loading /Users/damian/.config/GNS3/GNS3.ini.

We have 2 configs?

m00m00:~ damian$ ls /Users/damian/.config/GNS3/
GNS3.ini          base_configs/     exceptions.log    gns3_gui.log      gns3_server.log
GNS3_client.log   exception.log     gns3_gui.conf     gns3_server.conf
m00m00:~ damian$ ls /Users/damian/.config/GNS3/
gns3_gui.conf        gns3_gui.pid        gns3_server.log
gns3_gui.log        gns3_server.conf

Let's clear up these old config entries and start with a clean setup

m00m00:~ damian$ rm -rf ~/.config/gns3.net ~/.config/GNS3

And it works!

Perhaps this was handled in an intermediary release I missed, but it comes across as a pretty poor compatibility issue, especially considering the error only being visible on the command line.

A very quick look at upgrading CentOS 6.5 to 7.0

After some serious seeding action yesterday, today CentOS 7 was released!

As it happens I installed a new MSI Wind Box with CentOS 6.5 yesterday:

[root@pingu ~]# cat /etc/redhat-release
CentOS release 6.5 (Final)

[root@pingu ~]# uptime
 20:57:41 up 22:57,  1 user,  load average: 0.00, 0.04, 0.05

For the first time, there is a supported path for upgrading RHEL 6 to RHEL 7. Previously this would have been a reinstall.

Disclaimer: I would not attempt the following upgrade on a production server yet. The upgrade tools are still under development and not considered to be working properly (See the development mailing list post if you want to help out).

Since I have a new box that's not anywhere near production, what the hell... lets try this out.

First update to the latest CentOS 6 release

[root@pingu ~]# yum update
Loaded plugins: downloadonly, fastestmirror, security
Loading mirror speeds from cached hostfile
 * base: mirrors.coreix.net
 * epel: mirrors.mit.edu
 * extras: mirrors.coreix.net
 * updates: mirrors.clouvider.net
Setting up Update Process
Resolving Dependencies
--> Running transaction check
---> Package initscripts.x86_64 0:9.03.40-2.el6.centos.1 will be updated
---> Package initscripts.x86_64 0:9.03.40-2.el6.centos.2 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================================================================
 Package                      Arch                    Version                                 Repository                Size
=============================================================================================================================
Updating:
 initscripts                  x86_64                  9.03.40-2.el6.centos.2                  updates                  940 k

Transaction Summary
=============================================================================================================================
Upgrade       1 Package(s)

Total download size: 940 k
Is this ok [y/N]: y
Downloading Packages:
initscripts-9.03.40-2.el6.centos.2.x86_64.rpm                                                         | 940 kB     00:00     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Updating   : initscripts-9.03.40-2.el6.centos.2.x86_64                                                                 1/2 
Jul  8 20:58:08 pingu yum[6374]: Updated: initscripts-9.03.40-2.el6.centos.2.x86_64
  Cleanup    : initscripts-9.03.40-2.el6.centos.1.x86_64                                                                 2/2 
  Verifying  : initscripts-9.03.40-2.el6.centos.2.x86_64                                                                 1/2 
  Verifying  : initscripts-9.03.40-2.el6.centos.1.x86_64                                                                 2/2 

Updated:
  initscripts.x86_64 0:9.03.40-2.el6.centos.2                                                                                

Complete!

[root@pingu ~]# yum upgrade
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.clouvider.net
 * epel: mirror.vorboss.net
 * extras: centos.hyve.com
 * updates: centos.hyve.com
No packages marked for update

Install the upgrade utils

These are still very much in dev and may explode all the things. See http://lists.centos.org/pipermail/centos-devel/2014-July/011277.html for details.

[root@pingu ~]# yum localinstall http://dev.centos.org/centos/6/upg/x86_64/Packages/preupgrade-assistant-1.0.2-33.el6.x86_64.rpm http://dev.centos.org/centos/6/upg/x86_64/Packages/preupgrade-assistant-contents-0.5.13-1.el6.noarch.rpm http://dev.centos.org/centos/6/upg/x86_64/Packages/python-rhsm-1.9.7-1.el6.x86_64.rpm http://dev.centos.org/centos/6/upg/x86_64/Packages/redhat-upgrade-tool-0.7.22-1.el6.noarch.rpm
Loaded plugins: downloadonly, fastestmirror, security
Setting up Local Package Process
preupgrade-assistant-1.0.2-33.el6.x86_64.rpm                                                          | 438 kB     00:01     
Examining /var/tmp/yum-root-1lAuZF/preupgrade-assistant-1.0.2-33.el6.x86_64.rpm: preupgrade-assistant-1.0.2-33.el6.x86_64
Marking /var/tmp/yum-root-1lAuZF/preupgrade-assistant-1.0.2-33.el6.x86_64.rpm to be installed
Loading mirror speeds from cached hostfile
 * base: mirrors.coreix.net
 * epel: epel.mirror.constant.com
 * extras: mirrors.coreix.net
 * updates: mirrors.clouvider.net
preupgrade-assistant-contents-0.5.13-1.el6.noarch.rpm                                                 | 588 kB     00:01     
Examining /var/tmp/yum-root-1lAuZF/preupgrade-assistant-contents-0.5.13-1.el6.noarch.rpm: preupgrade-assistant-contents-0.5.13-1.el6.noarch
Marking /var/tmp/yum-root-1lAuZF/preupgrade-assistant-contents-0.5.13-1.el6.noarch.rpm to be installed
python-rhsm-1.9.7-1.el6.x86_64.rpm                                                                    |  99 kB     00:00     
Examining /var/tmp/yum-root-1lAuZF/python-rhsm-1.9.7-1.el6.x86_64.rpm: python-rhsm-1.9.7-1.el6.x86_64
Marking /var/tmp/yum-root-1lAuZF/python-rhsm-1.9.7-1.el6.x86_64.rpm to be installed
redhat-upgrade-tool-0.7.22-1.el6.noarch.rpm                                                           |  84 kB     00:00     
Examining /var/tmp/yum-root-1lAuZF/redhat-upgrade-tool-0.7.22-1.el6.noarch.rpm: 1:redhat-upgrade-tool-0.7.22-1.el6.noarch
Marking /var/tmp/yum-root-1lAuZF/redhat-upgrade-tool-0.7.22-1.el6.noarch.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package preupgrade-assistant.x86_64 0:1.0.2-33.el6 will be installed
--> Processing Dependency: openscap(x86-64) >= 0.9.3-1 for package: preupgrade-assistant-1.0.2-33.el6.x86_64
--> Processing Dependency: pkgconfig(libpcre) for package: preupgrade-assistant-1.0.2-33.el6.x86_64
--> Processing Dependency: pkgconfig(libxml-2.0) for package: preupgrade-assistant-1.0.2-33.el6.x86_64
--> Processing Dependency: pkgconfig(libxslt) for package: preupgrade-assistant-1.0.2-33.el6.x86_64
---> Package preupgrade-assistant-contents.noarch 0:0.5.13-1.el6 will be installed
---> Package python-rhsm.x86_64 0:1.9.7-1.el6 will be installed
--> Processing Dependency: m2crypto for package: python-rhsm-1.9.7-1.el6.x86_64
--> Processing Dependency: python-simplejson for package: python-rhsm-1.9.7-1.el6.x86_64
---> Package redhat-upgrade-tool.noarch 1:0.7.22-1.el6 will be installed
--> Running transaction check
---> Package libxml2-devel.x86_64 0:2.7.6-14.el6_5.2 will be installed
--> Processing Dependency: zlib-devel for package: libxml2-devel-2.7.6-14.el6_5.2.x86_64
---> Package libxslt-devel.x86_64 0:1.1.26-2.el6_3.1 will be installed
--> Processing Dependency: libgcrypt-devel for package: libxslt-devel-1.1.26-2.el6_3.1.x86_64
---> Package m2crypto.x86_64 0:0.20.2-9.el6 will be installed
---> Package openscap.x86_64 0:1.0.8-1.el6_5 will be installed
---> Package pcre-devel.x86_64 0:7.8-6.el6 will be installed
---> Package python-simplejson.x86_64 0:2.0.9-3.1.el6 will be installed
--> Running transaction check
---> Package libgcrypt-devel.x86_64 0:1.4.5-11.el6_4 will be installed
--> Processing Dependency: libgpg-error-devel for package: libgcrypt-devel-1.4.5-11.el6_4.x86_64
---> Package zlib-devel.x86_64 0:1.2.3-29.el6 will be installed
--> Running transaction check
---> Package libgpg-error-devel.x86_64 0:1.7-4.el6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================================================================
 Package                          Arch      Version              Repository                                             Size
=============================================================================================================================
Installing:
 preupgrade-assistant             x86_64    1.0.2-33.el6         /preupgrade-assistant-1.0.2-33.el6.x86_64             1.6 M
 preupgrade-assistant-contents    noarch    0.5.13-1.el6         /preupgrade-assistant-contents-0.5.13-1.el6.noarch    4.1 M
 python-rhsm                      x86_64    1.9.7-1.el6          /python-rhsm-1.9.7-1.el6.x86_64                       298 k
 redhat-upgrade-tool              noarch    1:0.7.22-1.el6       /redhat-upgrade-tool-0.7.22-1.el6.noarch              254 k
Installing for dependencies:
 libgcrypt-devel                  x86_64    1.4.5-11.el6_4       base                                                  118 k
 libgpg-error-devel               x86_64    1.7-4.el6            base                                                   14 k
 libxml2-devel                    x86_64    2.7.6-14.el6_5.2     updates                                               1.1 M
 libxslt-devel                    x86_64    1.1.26-2.el6_3.1     base                                                  561 k
 m2crypto                         x86_64    0.20.2-9.el6         base                                                  471 k
 openscap                         x86_64    1.0.8-1.el6_5        updates                                               2.9 M
 pcre-devel                       x86_64    7.8-6.el6            base                                                  318 k
 python-simplejson                x86_64    2.0.9-3.1.el6        base                                                  126 k
 zlib-devel                       x86_64    1.2.3-29.el6         base                                                   44 k

Transaction Summary
=============================================================================================================================
Install      13 Package(s)

Total size: 12 M
Total download size: 5.6 M
Installed size: 64 M
Is this ok [y/N]: y
Downloading Packages:
(1/9): libgcrypt-devel-1.4.5-11.el6_4.x86_64.rpm                                                      | 118 kB     00:00     
(2/9): libgpg-error-devel-1.7-4.el6.x86_64.rpm                                                        |  14 kB     00:00     
(3/9): libxml2-devel-2.7.6-14.el6_5.2.x86_64.rpm                                                      | 1.1 MB     00:00     
(4/9): libxslt-devel-1.1.26-2.el6_3.1.x86_64.rpm                                                      | 561 kB     00:00     
(5/9): m2crypto-0.20.2-9.el6.x86_64.rpm                                                               | 471 kB     00:00     
(6/9): openscap-1.0.8-1.el6_5.x86_64.rpm                                                              | 2.9 MB     00:01     
(7/9): pcre-devel-7.8-6.el6.x86_64.rpm                                                                | 318 kB     00:00     
(8/9): python-simplejson-2.0.9-3.1.el6.x86_64.rpm                                                     | 126 kB     00:00     
(9/9): zlib-devel-1.2.3-29.el6.x86_64.rpm                                                             |  44 kB     00:00     
-----------------------------------------------------------------------------------------------------------------------------
Total                                                                                        2.1 MB/s | 5.6 MB     00:02     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : zlib-devel-1.2.3-29.el6.x86_64                                                                           1/13 
  Installing : libxml2-devel-2.7.6-14.el6_5.2.x86_64                                                                    2/13 
  Installing : libgpg-error-devel-1.7-4.el6.x86_64                                                                      3/13 
  Installing : libgcrypt-devel-1.4.5-11.el6_4.x86_64                                                                    4/13 
  Installing : libxslt-devel-1.1.26-2.el6_3.1.x86_64                                                                    5/13 
  Installing : openscap-1.0.8-1.el6_5.x86_64                                                                            6/13 
  Installing : m2crypto-0.20.2-9.el6.x86_64                                                                             7/13 
  Installing : python-simplejson-2.0.9-3.1.el6.x86_64                                                                   8/13 
  Installing : python-rhsm-1.9.7-1.el6.x86_64                                                                           9/13 
  Installing : pcre-devel-7.8-6.el6.x86_64                                                                             10/13 
  Installing : preupgrade-assistant-1.0.2-33.el6.x86_64                                                                11/13 
  Installing : 1:redhat-upgrade-tool-0.7.22-1.el6.noarch                                                               12/13 
  Installing : preupgrade-assistant-contents-0.5.13-1.el6.noarch                                                       13/13 
  Verifying  : libgcrypt-devel-1.4.5-11.el6_4.x86_64                                                                    1/13 
  Verifying  : pcre-devel-7.8-6.el6.x86_64                                                                              2/13 
  Verifying  : python-simplejson-2.0.9-3.1.el6.x86_64                                                                   3/13 
  Verifying  : python-rhsm-1.9.7-1.el6.x86_64                                                                           4/13 
  Verifying  : 1:redhat-upgrade-tool-0.7.22-1.el6.noarch                                                                5/13 
  Verifying  : preupgrade-assistant-contents-0.5.13-1.el6.noarch                                                        6/13 
  Verifying  : m2crypto-0.20.2-9.el6.x86_64                                                                             7/13 
  Verifying  : openscap-1.0.8-1.el6_5.x86_64                                                                            8/13 
  Verifying  : libgpg-error-devel-1.7-4.el6.x86_64                                                                      9/13 
  Verifying  : libxml2-devel-2.7.6-14.el6_5.2.x86_64                                                                   10/13 
  Verifying  : libxslt-devel-1.1.26-2.el6_3.1.x86_64                                                                   11/13 
  Verifying  : zlib-devel-1.2.3-29.el6.x86_64                                                                          12/13 
  Verifying  : preupgrade-assistant-1.0.2-33.el6.x86_64                                                                13/13 

Installed:
  preupgrade-assistant.x86_64 0:1.0.2-33.el6               preupgrade-assistant-contents.noarch 0:0.5.13-1.el6              
  python-rhsm.x86_64 0:1.9.7-1.el6                         redhat-upgrade-tool.noarch 1:0.7.22-1.el6                        

Dependency Installed:
  libgcrypt-devel.x86_64 0:1.4.5-11.el6_4  libgpg-error-devel.x86_64 0:1.7-4.el6     libxml2-devel.x86_64 0:2.7.6-14.el6_5.2 
  libxslt-devel.x86_64 0:1.1.26-2.el6_3.1  m2crypto.x86_64 0:0.20.2-9.el6            openscap.x86_64 0:1.0.8-1.el6_5         
  pcre-devel.x86_64 0:7.8-6.el6            python-simplejson.x86_64 0:2.0.9-3.1.el6  zlib-devel.x86_64 0:1.2.3-29.el6        

Complete!

Check for potential problems

[root@pingu ~]# preupg
Preupg tool doesn't do the actual upgrade.
Please ensure you have backed up your system and/or data in the event of a failed upgrade
 that would require a full re-install of the system from installation media.
Do you want to continue? y/n
y
Gathering logs used by preupgrade assistant:
All installed packages : 01/10 ...finished (time 00:00s)
All changed files      : 02/10 ...finished (time 01:09s)
Changed config files   : 03/10 ...finished (time 00:00s)
All users              : 04/10 ...finished (time 00:00s)
All groups             : 05/10 ...finished (time 00:00s)
Service statuses       : 06/10 ...finished (time 00:00s)
All installed files    : 07/10 ...finished (time 00:01s)
All local files        : 08/10 ...finished (time 00:01s)
All executable files   : 09/10 ...finished (time 00:00s)
RedHat signed packages : 10/10 ...finished (time 00:00s)
Assessment of the system, running checks / SCE scripts:
001/100 ...done    (Configuration Files to Review)
002/100 ...done    (File Lists for Manual Migration)
003/100 ...done    (Bacula Backup Software)
004/100 ...done    (MySQL configuration)
005/100 ...done    (Migration of the MySQL data stack)
006/100 ...done    (General changes in default MySQL implementation)
007/100 ...done    (PostgreSQL upgrade content)
008/100 ...done    (GNOME Desktop Environment underwent several design modifications in Red Hat Enterprise Linux 7 release)
009/100 ...done    (KDE Desktop Environment underwent several design modifications in Red Hat Enterprise Linux 7 release)
010/100 ...done    (several graphic drivers not supported in Red Hat Enterprise Linux 7)
011/100 ...done    (several input drivers not supported in Red Hat Enterprise Linux 7)
012/100 ...done    (several kernel networking drivers not available in Red Hat Enterprise Linux 7)
013/100 ...done    (several kernel storage drivers not available in Red Hat Enterprise Linux 7)
014/100 ...done    (Names, Options and Output Format Changes in arptables)
015/100 ...done    (BIND9 running in a chroot environment check.)
016/100 ...done    (BIND9 configuration compatibility check)
017/100 ...done    (Move dhcpd/dhcprelay arguments from /etc/sysconfig/* to *.service files)
018/100 ...done    (DNSMASQ configuration compatibility check)
019/100 ...done    (Dovecot configuration compatibility check)
020/100 ...done    (Compatibility Between iptables and ip6tables)
021/100 ...done    (Net-SNMP check)
022/100 ...done    (Squid configuration compatibility check)
023/100 ...done    (Reusable Configuration Files)
024/100 ...done    (VCS repositories)
025/100 ...done    (Added and extended options for BIND9 configuration)
026/100 ...done    (Added options in DNSMASQ configuration)
027/100 ...done    (Packages not signed by Red Hat)
028/100 ...done    (Obsoleted rpms)
029/100 ...done    (w3m not available in Red Hat Enterprise Linux 7)
030/100 ...done    (report incompatibilities between Red Hat Enterprise Linux 6 and 7 in qemu-guest-agent package)
031/100 ...done    (Removed options in coreutils binaries)
032/100 ...done    (Removed options in gawk binaries)
033/100 ...done    (Removed options in netstat binary)
034/100 ...done    (Removed options in quota tools)
035/100 ...done    (Removed rpms)
036/100 ...done    (Replaced rpms)
037/100 ...done    (GMP library incompatibilities)
038/100 ...done    (optional channel problems)
039/100 ...done    (package downgrades)
040/100 ...done    (restore custom selinux configuration)
041/100 ...done    (General)
042/100 ...done    (samba shared directories selinux)
043/100 ...done    (CUPS Browsing/BrowsePoll configuration)
044/100 ...done    (CVS Package Split)
045/100 ...done    (FreeRADIUS Upgrade Verification)
046/100 ...done    (httpd configuration compatibility check)
047/100 ...done    (bind-dyndb-ldap)
048/100 ...done    (Identity Management Server compatibility check)
049/100 ...done    (IPA Server CA Verification)
050/100 ...done    (NTP configuration)
051/100 ...done    (Information on time-sync.target)
052/100 ...done    (OpenLDAP /etc/sysconfig and data compatibility)
053/100 ...done    (OpenSSH sshd_config migration content)
054/100 ...done    (OpenSSH sysconfig migration content)
055/100 ...done    (Configuration for quota_nld service)
056/100 ...done    (Disk quota netlink message daemon moved into quota-nld package)
057/100 ...done    (SSSD compatibility check)
058/100 ...done    (Luks encrypted partition)
059/100 ...done    (Clvmd and cmirrord daemon management.)
060/100 ...done    (State of LVM2 services.)
061/100 ...done    (device-mapper-multipath configuration compatibility check)
062/100 ...done    (Removal of scsi-target-utils)
063/100 ...done    (Configuration for warnquota tool)
064/100 ...done    (Disk quota tool warnquota moved into quota-warnquota package)
065/100 ...done    (Check for Add-On availability)
066/100 ...done    (Architecture Support)
067/100 ...done    (Binary rebuilds)
068/100 ...done    (Debuginfo packages)
069/100 ...done    (Cluster and High Availablility)
070/100 ...done    (fix krb5kdc config file)
071/100 ...done    (File Systems, Partitions and Mounts Configuration Review)
072/100 ...done    (Read Only FHS directories)
073/100 ...done    (Red Hat Enterprise Linux Server variant)
074/100 ...done    (Sonamebumped libs)
075/100 ...done    (SonameKept Reusable Dynamic Libraries)
076/100 ...done    (Removed .so libs)
077/100 ...done    (In-place Upgrade Requirements for the /usr/ Directory)
078/100 ...done    (CA certificate bundles modified)
079/100 ...done    (Developer Tool Set packages)
080/100 ...done    (Hyper-V)
081/100 ...done    (Content for enabling and disabling services based on RHEL 6 system)
082/100 ...done    (Check for ethernet interface naming)
083/100 ...done    (User modification in /etc/rc.local and /etc/rc.d/rc.local)
084/100 ...done    (cgroups configuration compatibility check)
085/100 ...done    (Plugable authentication modules (PAM))
086/100 ...done    (Foreign Perl modules)
087/100 ...done    (Python 2.7.5)
088/100 ...done    (Ruby 2.0.0)
089/100 ...done    (SCL collections)
090/100 ...done    (Red Hat Subscription Manager)
091/100 ...done    (Red Hat Network Classic Unsupported)
092/100 ...done    (System kickstart)
093/100 ...done    (YUM)
094/100 ...done    (Check for usage of dangerous range of UID/GIDs)
095/100 ...done    (Incorrect usage of reserved UID/GIDs)
096/100 ...done    (NIS ypbind config files back-up)
097/100 ...done    (NIS Makefile back-up)
098/100 ...done    (NIS server maps check)
099/100 ...done    (NIS server MAXUID and MAXGID limits check)
100/100 ...done    (NIS server config file back-up)
Assessment finished (time 00:00s)
Result table with checks and their results for main contents:
------------------------------------------------------------------------------------------------------------------------------
|Configuration Files to Review                                                                               |notapplicable  |
|File Lists for Manual Migration                                                                             |notapplicable  |
|Bacula Backup Software                                                                                      |notapplicable  |
|MySQL configuration                                                                                         |notapplicable  |
|Migration of the MySQL data stack                                                                           |notapplicable  |
|General changes in default MySQL implementation                                                             |notapplicable  |
|PostgreSQL upgrade content                                                                                  |notapplicable  |
|GNOME Desktop Environment underwent several design modifications in Red Hat Enterprise Linux 7 release      |notapplicable  |
|KDE Desktop Environment underwent several design modifications in Red Hat Enterprise Linux 7 release        |notapplicable  |
|several graphic drivers not supported in Red Hat Enterprise Linux 7                                         |notapplicable  |
|several input drivers not supported in Red Hat Enterprise Linux 7                                           |notapplicable  |
|several kernel networking drivers not available in Red Hat Enterprise Linux 7                               |notapplicable  |
|several kernel storage drivers not available in Red Hat Enterprise Linux 7                                  |notapplicable  |
|Names, Options and Output Format Changes in arptables                                                       |notapplicable  |
|BIND9 running in a chroot environment check.                                                                |notapplicable  |
|BIND9 configuration compatibility check                                                                     |notapplicable  |
|Move dhcpd/dhcprelay arguments from /etc/sysconfig/* to *.service files                                     |notapplicable  |
|DNSMASQ configuration compatibility check                                                                   |notapplicable  |
|Dovecot configuration compatibility check                                                                   |notapplicable  |
|Compatibility Between iptables and ip6tables                                                                |notapplicable  |
|Net-SNMP check                                                                                              |notapplicable  |
|Squid configuration compatibility check                                                                     |notapplicable  |
|Reusable Configuration Files                                                                                |notapplicable  |
|VCS repositories                                                                                            |notapplicable  |
|Added and extended options for BIND9 configuration                                                          |notapplicable  |
|Added options in DNSMASQ configuration                                                                      |notapplicable  |
|Packages not signed by Red Hat                                                                              |notapplicable  |
|Obsoleted rpms                                                                                              |notapplicable  |
|w3m not available in Red Hat Enterprise Linux 7                                                             |notapplicable  |
|report incompatibilities between Red Hat Enterprise Linux 6 and 7 in qemu-guest-agent package               |notapplicable  |
|Removed options in coreutils binaries                                                                       |notapplicable  |
|Removed options in gawk binaries                                                                            |notapplicable  |
|Removed options in netstat binary                                                                           |notapplicable  |
|Removed options in quota tools                                                                              |notapplicable  |
|Removed rpms                                                                                                |notapplicable  |
|Replaced rpms                                                                                               |notapplicable  |
|GMP library incompatibilities                                                                               |notapplicable  |
|optional channel problems                                                                                   |notapplicable  |
|package downgrades                                                                                          |notapplicable  |
|restore custom selinux configuration                                                                        |notapplicable  |
|General                                                                                                     |notapplicable  |
|samba shared directories selinux                                                                            |notapplicable  |
|CUPS Browsing/BrowsePoll configuration                                                                      |notapplicable  |
|CVS Package Split                                                                                           |notapplicable  |
|FreeRADIUS Upgrade Verification                                                                             |notapplicable  |
|httpd configuration compatibility check                                                                     |notapplicable  |
|bind-dyndb-ldap                                                                                             |notapplicable  |
|Identity Management Server compatibility check                                                              |notapplicable  |
|IPA Server CA Verification                                                                                  |notapplicable  |
|NTP configuration                                                                                           |notapplicable  |
|Information on time-sync.target                                                                             |notapplicable  |
|OpenLDAP /etc/sysconfig and data compatibility                                                              |notapplicable  |
|OpenSSH sshd_config migration content                                                                       |notapplicable  |
|OpenSSH sysconfig migration content                                                                         |notapplicable  |
|Configuration for quota_nld service                                                                         |notapplicable  |
|Disk quota netlink message daemon moved into quota-nld package                                              |notapplicable  |
|SSSD compatibility check                                                                                    |notapplicable  |
|Luks encrypted partition                                                                                    |notapplicable  |
|Clvmd and cmirrord daemon management.                                                                       |notapplicable  |
|State of LVM2 services.                                                                                     |notapplicable  |
|device-mapper-multipath configuration compatibility check                                                   |notapplicable  |
|Removal of scsi-target-utils                                                                                |notapplicable  |
|Configuration for warnquota tool                                                                            |notapplicable  |
|Disk quota tool warnquota moved into quota-warnquota package                                                |notapplicable  |
|Check for Add-On availability                                                                               |notapplicable  |
|Architecture Support                                                                                        |notapplicable  |
|Binary rebuilds                                                                                             |notapplicable  |
|Debuginfo packages                                                                                          |notapplicable  |
|Cluster and High Availablility                                                                              |notapplicable  |
|fix krb5kdc config file                                                                                     |notapplicable  |
|File Systems, Partitions and Mounts Configuration Review                                                    |notapplicable  |
|Read Only FHS directories                                                                                   |notapplicable  |
|Red Hat Enterprise Linux Server variant                                                                     |notapplicable  |
|Sonamebumped libs                                                                                           |notapplicable  |
|SonameKept Reusable Dynamic Libraries                                                                       |notapplicable  |
|Removed .so libs                                                                                            |notapplicable  |
|In-place Upgrade Requirements for the /usr/ Directory                                                       |notapplicable  |
|CA certificate bundles modified                                                                             |notapplicable  |
|Developer Tool Set packages                                                                                 |notapplicable  |
|Hyper-V                                                                                                     |notapplicable  |
|Content for enabling and disabling services based on RHEL 6 system                                          |notapplicable  |
|Check for ethernet interface naming                                                                         |notapplicable  |
|User modification in /etc/rc.local and /etc/rc.d/rc.local                                                   |notapplicable  |
|cgroups configuration compatibility check                                                                   |notapplicable  |
|Plugable authentication modules (PAM)                                                                       |notapplicable  |
|Foreign Perl modules                                                                                        |notapplicable  |
|Python 2.7.5                                                                                                |notapplicable  |
|Ruby 2.0.0                                                                                                  |notapplicable  |
|SCL collections                                                                                             |notapplicable  |
|Red Hat Network Classic Unsupported                                                                         |notapplicable  |
|Red Hat Subscription Manager                                                                                |notapplicable  |
|System kickstart                                                                                            |notapplicable  |
|YUM                                                                                                         |notapplicable  |
|Check for usage of dangerous range of UID/GIDs                                                              |notapplicable  |
|Incorrect usage of reserved UID/GIDs                                                                        |notapplicable  |
|NIS ypbind config files back-up                                                                             |notapplicable  |
|NIS Makefile back-up                                                                                        |notapplicable  |
|NIS server maps check                                                                                       |notapplicable  |
|NIS server MAXUID and MAXGID limits check                                                                   |notapplicable  |
|NIS server config file back-up                                                                              |notapplicable  |
------------------------------------------------------------------------------------------------------------------------------
Tarball with results is stored here /root/preupgrade-results/preupg_results-140708210602.tar.gz .
The latest assessment is stored in directory /root/preupgrade .
Upload results to UI by command:
e.g. preupg -u http://127.0.0.1:8099/submit/ -r /root/preupgrade-results/preupg_results-*.tar.gz .

Hopefully everything looks ok and you can continue with the upgrade.

Run the upgrade

This tool basically downloads a custom kernel which does the actual upgrade.

This should work

redhat-upgrade-tool-cli http://mirror.bytemark.co.uk/centos/7/os/x86_64/

However it fails as below

[root@pingu ~]# redhat-upgrade-tool-cli --instrepo=http://mirror.bytemark.co.uk/centos/7/os/x86_64/ --network=7 --force --disablerepo=epel
setting up repos...
No upgrade available for the following repos: scl
.treeinfo                                                                                                                        | 1.1 kB     00:00     
getting boot images...
setting up update...
verify local files 100% [==============================================================================================================================]
(1/5): libdwarf-20130207-3.el7.x86_64.rpm                                                                                        | 109 kB     00:00     
(2/5): python-dns-1.10.0-5.el7.noarch.rpm                                                                                        | 220 kB     00:00     
(3/5): qpdf-libs-5.0.1-3.el7.x86_64.rpm                                                                                          | 328 kB     00:00     
(4/5): qrencode-libs-3.4.1-3.el7.x86_64.rpm                                                                                      |  50 kB     00:00     
(5/5): satyr-0.13-4.el7.x86_64.rpm                                                                                               | 500 kB     00:00     
warning: rpmts_HdrFromFdno: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY

Downloading failed: The GPG keys listed for the "CentOS-7.0 - Base" repository are already installed but they are not correct for this package.
Check that the correct key URLs are configured for this repository.

So I thought.... ok, for now we'll trust the packages and not check the GPG keys. However there also seems to be a problem in detecting preupg. We ran preupg first, to check the system however the upgrade tool thinks we havn't and fails as below

[root@pingu ~]# redhat-upgrade-tool-cli --instrepo=http://mirror.bytemark.co.uk/centos/7/os/x86_64/ --network=7 --nogpgcheck
setting up repos...
base                                                                                                                             | 3.6 kB     00:00     
base/primary_db                                                                                                                  | 4.9 MB     00:01     
cmdline-instrepo                                                                                                                 | 3.6 kB     00:00     
cmdline-instrepo/primary_db                                                                                                      | 4.9 MB     00:01     
epel/metalink                                                                                                                    |  26 kB     00:00     
epel                                                                                                                             | 4.4 kB     00:00     
epel/primary_db                                                                                                                  | 6.2 MB     00:03     
extras                                                                                                                           | 2.9 kB     00:00     
extras/primary_db                                                                                                                |  15 kB     00:00     
updates                                                                                                                          | 2.9 kB     00:00     
updates/primary_db                                                                                                               | 1.4 MB     00:00     
No upgrade available for the following repos: scl
.treeinfo                                                                                                                        | 1.1 kB     00:00     
preupgrade-assistant has not been run.
To perform this upgrade, either run preupg or run redhat-upgrade-tool --force

I finally ended up disabling both GPG encryption and forcing the run (to skip the preupg)

[root@pingu ~]# redhat-upgrade-tool-cli --instrepo=http://mirror.bytemark.co.uk/centos/7/os/x86_64/ --network=7 --force --nogpgcheck
setting up repos...
No upgrade available for the following repos: scl
.treeinfo                                                                                                                        | 1.1 kB     00:00     
getting boot images...
setting up update...
verify local files 100% [==============================================================================================================================]
redhat_upgrade_tool.yum WARNING: Error loading productid metadata for base.
redhat_upgrade_tool.yum WARNING: Error loading productid metadata for epel.
redhat_upgrade_tool.yum WARNING: Error loading productid metadata for extras.
redhat_upgrade_tool.yum WARNING: Error loading productid metadata for updates.
testing upgrade transaction
rpm transaction 100% [=================================================================================================================================]
rpm install 100% [=====================================================================================================================================]
setting up system for upgrade
Finished. Reboot to start upgrade.

Booom - we're getting somewhere. Now we reboot into the kernel that has been installed.

[root@pingu ~]# reboot

At this point go get a cup of tea or something - the upgrade kernel basically downloads all the required packages from yum and then reboots into a working system.

The system booted up!

Since there isn't a SCL repo for 7 yet - we need to clean this up

[root@pingu ~]# rm -f /etc/yum.repos.d/CentOS-SCL.repo

I also have EPEL installed on this server - lets fix that up.....

[root@pingu ~]# wget -O /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 http://mirror.bytemark.co.uk/fedora/epel/RPM-GPG-KEY-EPEL-7
[root@pingu ~]# sed -i 's/6/7/g' /etc/yum.repos.d/epel.repo 

Update packages

At this point CentOS 7 should update cleanly

[root@pingu ~]# yum update
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.clouvider.net
 * epel: mirror.vorboss.net
 * extras: centos.hyve.com
 * updates: centos.hyve.com
Resolving Dependencies
--> Running transaction check
---> Package epel-release.noarch 0:6-8 will be updated
---> Package epel-release.noarch 0:7-0.2 will be an update
---> Package htop.x86_64 0:1.0.1-2.el6 will be updated
---> Package htop.x86_64 0:1.0.3-3.el7 will be an update
---> Package python-crypto.x86_64 0:2.0.1-22.el6 will be updated
---> Package python-crypto.x86_64 0:2.6.1-1.el7 will be an update
---> Package python-paramiko.noarch 0:1.7.5-2.1.el6 will be updated
---> Package python-paramiko.noarch 0:1.11.3-1.el7 will be an update
---> Package python-simplejson.x86_64 0:2.0.9-3.1.el6 will be updated
---> Package python-simplejson.x86_64 0:3.3.3-1.el7 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================================================================================================
 Package                                    Arch                            Version                                 Repository                     Size
========================================================================================================================================================
Updating:
 epel-release                               noarch                          7-0.2                                   epel                           13 k
 htop                                       x86_64                          1.0.3-3.el7                             epel                           87 k
 python-crypto                              x86_64                          2.6.1-1.el7                             epel                          469 k
 python-paramiko                            noarch                          1.11.3-1.el7                            epel                          678 k
 python-simplejson                          x86_64                          3.3.3-1.el7                             epel                          171 k

Transaction Summary
========================================================================================================================================================
Upgrade  5 Packages

Total size: 1.4 M
Is this ok [y/d/N]: y
Downloading packages:
warning: /var/cache/yum/x86_64/7/epel/packages/epel-release-7-0.2.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 352c64e5: NOKEY
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
Importing GPG key 0x352C64E5:
 Userid     : "Fedora EPEL (7) <epel@fedoraproject.org>"
 Fingerprint: 91e9 7d7c 4a5e 96f1 7f3e 888f 6a2f aea2 352c 64e5
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
Is this ok [y/N]: y
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Warning: RPMDB altered outside of yum.
** Found 20 pre-existing rpmdb problem(s), 'yum check' output follows:
aiccu-2007.01.15-7.el6.x86_64 has missing requires of libgnutls.so.26()(64bit)
aiccu-2007.01.15-7.el6.x86_64 has missing requires of libgnutls.so.26(GNUTLS_1_4)(64bit)
cas-0.15-1.el6.1.noarch has missing requires of python(abi) = ('0', '2.6', None)
10:centos-release-SCL-6-5.el6.centos.x86_64 has missing requires of centos-release = ('0', '6', None)
cloog-ppl-0.15.7-1.2.el6.x86_64 has missing requires of libgmp.so.3()(64bit)
hal-info-20090716-3.1.el6.noarch has missing requires of hal >= ('0', '0.5.10', None)
openscap-1.0.8-1.el6_5.x86_64 has missing requires of libpcre.so.0()(64bit)
openscap-1.0.8-1.el6_5.x86_64 has missing requires of librpm.so.1()(64bit)
openscap-1.0.8-1.el6_5.x86_64 has missing requires of librpmio.so.1()(64bit)
ppl-0.10.2-11.el6.x86_64 has missing requires of libgmp.so.3()(64bit)
preupgrade-assistant-1.0.2-33.el6.x86_64 has missing requires of libpcre.so.0()(64bit)
preupgrade-assistant-1.0.2-33.el6.x86_64 has missing requires of librpm.so.1()(64bit)
preupgrade-assistant-1.0.2-33.el6.x86_64 has missing requires of librpmio.so.1()(64bit)
preupgrade-assistant-1.0.2-33.el6.x86_64 has missing requires of python(abi) = ('0', '2.6', None)
python-dns-1.11.1-2.el6.noarch has missing requires of python(abi) = ('0', '2.6', None)
python-iwlib-0.1-1.2.el6.x86_64 has missing requires of libpython2.6.so.1.0()(64bit)
python-iwlib-0.1-1.2.el6.x86_64 has missing requires of python(abi) = ('0', '2.6', None)
qpdf-libs-5.1.1-2.el6.x86_64 has missing requires of libpcre.so.0()(64bit)
1:readahead-1.5.6-2.el6.x86_64 has missing requires of upstart
satyr-0.14-1.el6.x86_64 has missing requires of librpm.so.1()(64bit)
  Updating   : python-crypto-2.6.1-1.el7.x86_64                                                                                                    1/10 
  Updating   : python-paramiko-1.11.3-1.el7.noarch                                                                                                 2/10 
  Updating   : python-simplejson-3.3.3-1.el7.x86_64                                                                                                3/10 
  Updating   : htop-1.0.3-3.el7.x86_64                                                                                                             4/10 
  Updating   : epel-release-7-0.2.noarch                                                                                                           5/10 
warning: /etc/yum.repos.d/epel.repo saved as /etc/yum.repos.d/epel.repo.rpmsave
  Cleanup    : python-paramiko-1.7.5-2.1.el6.noarch                                                                                                6/10 
  Cleanup    : epel-release-6-8.noarch                                                                                                             7/10 
  Cleanup    : python-crypto-2.0.1-22.el6.x86_64                                                                                                   8/10 
  Cleanup    : python-simplejson-2.0.9-3.1.el6.x86_64                                                                                              9/10 
  Cleanup    : htop-1.0.1-2.el6.x86_64                                                                                                            10/10 
  Verifying  : epel-release-7-0.2.noarch                                                                                                           1/10 
  Verifying  : python-crypto-2.6.1-1.el7.x86_64                                                                                                    2/10 
  Verifying  : htop-1.0.3-3.el7.x86_64                                                                                                             3/10 
  Verifying  : python-simplejson-3.3.3-1.el7.x86_64                                                                                                4/10 
  Verifying  : python-paramiko-1.11.3-1.el7.noarch                                                                                                 5/10 
  Verifying  : python-simplejson-2.0.9-3.1.el6.x86_64                                                                                              6/10 
  Verifying  : epel-release-6-8.noarch                                                                                                             7/10 
  Verifying  : python-crypto-2.0.1-22.el6.x86_64                                                                                                   8/10 
  Verifying  : python-paramiko-1.7.5-2.1.el6.noarch                                                                                                9/10 
  Verifying  : htop-1.0.1-2.el6.x86_64                                                                                                            10/10 

Updated:
  epel-release.noarch 0:7-0.2               htop.x86_64 0:1.0.3-3.el7    python-crypto.x86_64 0:2.6.1-1.el7    python-paramiko.noarch 0:1.11.3-1.el7   
  python-simplejson.x86_64 0:3.3.3-1.el7   

Complete!

Let's reboot once more to make sure things are all cleanly started...

[root@pingu ~]# reboot

Oh hey, it's a CentOS 7 box

[root@pingu ~]# cat /etc/redhat-release 
CentOS Linux release 7.0.1406 (Core) 

Roundup

While it's too new to try out anywhere near production kit, for a set of tools that 'will need some patching to work properly' everything works pretty well.

Hopefully with a few minor tweaks the packages will land on the normal mirrors and upgrading boxes will be bliss.

Go play with CentOS 7 and try out some of the neat features in the 3.10 kernel.

As always, remember to leave SELinux turned on.

What's Anycast and why would I use it?

Recently I've heard some confusion as to what Anycast is, ranging from it being part of the IPv6 specification to multicast.

While I'm not a network dude, hopefully this post will give you an overview of what Anycast is, how it's useful and what to watch out for.

Overview

Firstly to clear up some confusion, Anycast is not a protocol (TCP/UDP) or protocol version (IPv4/IPv6), but a type of addressing such as Unicast or Broadcast.

As far as the 'client' is concerned, they are talking unicast (to a single node), but this may be routed to one of many nodes depending on the routing table.

A quick example of this happening can be seen if you traceroute to an Anycast IP address from different locations in the world as seen below

traceroute to ian.ns.cloudflare.com (173.245.59.118), 30 hops max, 60 byte packets
 1  router1 (10.44.200.254)  15.334 ms  17.279 ms  19.183 ms
 2  host-92-23-160-1.as13285.net (92.23.160.1)  29.204 ms  32.556 ms  36.016 ms
 3  host-78-151-225-101.static.as13285.net (78.151.225.101)  38.386 ms  40.704 ms  42.598 ms
 4  host-78-151-225-84.static.as13285.net (78.151.225.84)  63.415 ms xe-11-2-0-bragg002.bir.as13285.net (78.151.225.72)  44.347 ms  46.697 ms
 5  xe-11-1-0-rt001.the.as13285.net (62.24.240.6)  49.117 ms xe-11-1-0-rt001.sov.as13285.net (62.24.240.14)  51.454 ms  54.374 ms
 6  host-78-144-1-61.as13285.net (78.144.1.61)  56.102 ms  48.605 ms  41.366 ms
 7  host-78-144-0-180.as13285.net (78.144.0.180)  42.400 ms host-78-144-0-116.as13285.net (78.144.0.116)  39.599 ms host-78-144-0-164.as13285.net (78.144.0.164)  37.310 ms
 8  195.66.225.179 (195.66.225.179)  38.239 ms  73.818 ms  72.494 ms
 9  ian.ns.cloudflare.com (173.245.59.118)  38.989 ms  37.828 ms  38.971 ms
traceroute to ian.ns.cloudflare.com (173.245.59.118), 30 hops max, 60 byte packets
 1  router1-dal.linode.com (67.18.7.161)  19.229 ms  19.291 ms  19.392 ms
 2  xe-2-0-0.car03.dllstx2.networklayer.com (67.18.7.89)  0.197 ms  0.220 ms  0.202 ms
 3  po101.dsr01.dllstx2.networklayer.com (70.87.254.73)  0.513 ms  0.537 ms  0.624 ms
 4  po21.dsr01.dllstx3.networklayer.com (70.87.255.65)  0.902 ms  0.983 ms  1.020 ms
 5  ae16.bbr01.eq01.dal03.networklayer.com (173.192.18.224)  16.145 ms  16.138 ms  16.112 ms
 6  141.101.74.253 (141.101.74.253)  0.602 ms  0.566 ms  0.549 ms
 7  ian.ns.cloudflare.com (173.245.59.118)  0.523 ms  0.670 ms  0.493 ms

Note that the DNS (ian.ns.cloudflare.com) resolves to the same IP (173.245.59.118), but the traffic goes to different routers (141.101.74.253 vs 195.66.225.179).

The 'magic' here happens at the routing layer, a router will have multiple 'paths' to the IP (173.245.59.118) and chooses the best one based on its metrics.

A large (public) example of Anycast being used is the root nameservers - combined there are something like 13 (v4) IP addresses serving NS entries, with ~350 servers behind those 13 IPs distributed throughout the world.

For those used to dealing with internal networks

Imagine you have 3 offices configured in a L2 triangle for redundancy:

Your gateway is England, Scotland and Wales need to access it if either of their links fail. You also have servers in each of the offices and want to access them as fast as possible.

Keeping the links at layer 2 and blocking one (via something like STP) would give you gateway redundancy - if link A failed, link B would unblock and start forwarding traffic.

The problem here is you are having to go via Scotland to access Wales from England, when they have a direct link. You can't have both links up because you'd cause a loop and no traffic would pass.

The solution is to make both links layer 3 and use a routing protocol to advertise the ranges over the top of them - there are reasons you'd want to use layer 2, or have one layer 2 and one layer 3, this is outside the scope of this example though.

I'm not going to get into how to configure a routing protocol - there are a few to choose from, for this use case it doesn't hugely matter. Let's assume both links are the same speed and are direct (1 hop away).

The topology looks something like

This means if any of the offices want to talk to another office, they are only 1 hop away and don't have to go though a middle man gateway - but if the direct link fails, they can be re-routed the long (2 hops) way around.

In summary using routing between the 3 offices allows for transparent fail over and best-performance access. In a more complicated topology this is very useful.

Essentially, this is also how 'Anycast' works - the main difference being is we're dealing with multiple nodes, rather than just re-advertising a single node (route). The routing doesn't care either way - it will choose the 'best' path based on what it knows about.

Why is this useful?

Anycast gives you a number of benefits such as

  • Higher reliability/availability
  • Higher performance
  • Localisation/migration of DoS/DDoS attacks
  • Client agnostic

Why doesn't everyone use it?

There are some drawbacks to using Anycast addressing

  • Can be complex to deploy - requires an AS, addressing/routing management, more routes being advertised
  • Can be expensive - IP space, network infrastructure, training
  • Harder to troubleshoot
  • Harder to monitor

How would I use it?

One of the most common use in for DNS services, however any stateless protocol can take advantage of Anycast - it is possible to use stateful protocols, however you need a much higher level of control and risk 'breaking' connections across 2 nodes.

Imagine you where serving users out of Europe, US and Asia-Pacific.

A common thing to do would be create 3 NS entries pointing to servers in each location

myawesomedns.something. 360 IN  SOA ns1.myawesomedns.something. root.myawesomedns.something. (1303702991 14400 14400 1209600 86400)

myawesomedns.something. 360 IN  NS  ns1.myawesomedns.something.
myawesomedns.something. 360 IN  NS  ns2.myawesomedns.something.
myawesomedns.something. 360 IN  NS  ns3.myawesomedns.something.

; EU
ns1 360 IN  A   10.0.0.1

; APAC
ns2 360 IN  A   192.168.0.1

; US
ns3 360 IN  A   172.16.0.1

These NS records would then be "glued" on the root nameservers, so the first resolver can find the NS servers IP address.

Now there are 2 problems with this, if you loose a server the resolver (client) will be slow to fail over (probably timing out a few requests) and a user in the US might end up going to the server in APAC rather than the 'local' one to them.

Anycast helps to solve both these issues.

If a server goes down you can withdraw the route and (as long as you're not flapping routes) your peers should pick it up in a few minutes.

In the case of a user making a request to the Anycast IP, the request will be routed to the 'nearest' node - in normal operation this would be the 'local' one, if that had failed it would automatically (after the route withdrawal) go to the next 'nearest'.

Now I'm quoting 'nearest' and 'local' and these are determined by your routing protocol, normally based on metrics such as weight, hops or speed, rather than physical location.

The routing before we start using Anycast

Requests to ns{1..3} (192.168.0.1, 127.16.0.1, 10.0.0.1) before we start using Anycast

So lets start using Anycast, we're going to keep the original IPs as 'maintenance' IPs - ones we know only go to that single host and create a new block.

myawesomedns.something. 360 IN  SOA ns1.myawesomedns.something. root.myawesomedns.something. (1303702991 14400 14400 1209600 86400)

myawesomedns.something. 360 IN  NS  ns1.myawesomedns.something.
myawesomedns.something. 360 IN  NS  ns2.myawesomedns.something.
myawesomedns.something. 360 IN  NS  ns3.myawesomedns.something.

; EU
srv-ns1 360 IN  A   10.0.0.1

; APAC
srv-ns2 360 IN  A   192.168.0.1

; US
srv-ns3 360 IN  A   172.16.0.1

; Anycast
ns1 360 IN A 10.1.0.1
ns2 360 IN A 10.1.0.2
ns3 360 IN A 10.1.0.3

10.1.0.1, 10.1.0.2 & 10.1.0.3 now need to be advertised from the EU, APAC and US routers as /32 blocks - with no summary route for 10.1.0.0 0.0.0.255. 10.1.0.1, 10.1.0.2 & 10.1.0.3 also need to be brought up on loopback interfaces on the server and the DNS server configured to bind to them.

Once this propagates out routing will take care of sending your traffic to the right place, in the event of an outage or maintenance just withdraw the routes so they are no longer advertised and traffic will be re-routed.

This is slightly oversimplified as there is a certain level of tuning needed to stop flapping, which can cause your updates to be delayed (due to dampening timers etc).

I won't go into the BGP configuration in this post as it's a little lengthy to explain/draw how routing tables look, essentially you configure your neighbours and send them /32 routes - cisco have a nice example for configuring BGP on IOS. Be careful (use filters) when dealing with BGP/EGP protocols as you can do really weird things ;)

The routing after we start using 'Anycast' IPs

Requests to ns{1..3} (10.1.0.{1..3}) after we start using 'Anycast' IPs

In the case of the EU site failing the routes would withdraw and requests would be re-routed

In the case of the EU and the US sites failing, requests would be re-routed in the same manner

All this happens at layer 3 - in the DNS service scenario we would probably use GeoIP logic at layer 7 to localise the response and then serve the request.

Slow SSH to new CentOS servers

Recently when installing new servers I 'randomly' had an issue where SSH connections would lag out for 20-30 seconds on login then function fine once authenticated.

I noticed this issue on CentOS 5/6 servers, however it probably applies to anything running glibc 2.10+.

Now while there are a few reasons SSH might be doing this (most of which can be seen using verbose flags), it wasn't the usual suspects (GSSAPI etc).

The interesting part was, this wasn't happening on all servers (they are standard config wise thanks to puppet).

It turns out the UseDNS setting was failing, very slowly. Now a simple way to quickly solve the issue is to set UseDNS no in your sshd_config file.

This might be acceptable, depending on how much you have access restricted however there is an alternative.

It turns out there was a change in glibc that can be seen on the changelog here.

Under some circumstances (especially when firewalls are involved) DNS resolution will fail and glibc should fallback, close the socket and start again. It would seem this takes rather a long while when trying to SSH!

Thankfully there is a (recently documented) option that you can add to your resolv.conf file single-request-reopen. This forces glibc to enter fallback mode straight away and kills the delay.

An example of the 'patched' /etc/resolv.conf file

# This file is managed via Puppet

# Servers
nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 208.67.222.222
nameserver 208.67.220.220

# Options
options single-request-reopen

Now you might want to dig into your network stack a little more and make sure you can handle IPv6/dual stack properly but for now, problem solved!