Tech Notes :- ssh failure Offending RSA key in /var/lib/sss/pubconf/known_hosts

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:gEFs4gmKdgDsV2n3UhB/lNhZ/sLSM8q+5VIbl+D1S3c.
Please contact your system administrator.
Add correct host key in /dev/null to get rid of this message.
Offending RSA key in /var/lib/sss/pubconf/known_hosts:8
Password authentication is disabled to avoid man-in-the-middle attacks.
Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.

This is not a normal ssh error that can be fixed by editing ~/.ssh/known_hosts and removing the relevant key entries or by using the command ssh-keygen -f ~/.ssh/known_hosts -R <host>. If you try to do it, the key will come back and ends up with the same error.

Another option that we are tempted to try is to remove the offending key from /var/lib/sss/pubconf/known_hosts. This also will end in failure

Most probably this error occurs when the server was part of  IPA setup and you just re-installed the server without removing the relevant configuration from the IPA server.In this case IPA server will still have the keys associated with the server before it got re-installed and we have to first remove the keys from the IPA server.

You can do that from the IPA Identity Management Web UI by browsing to Hosts--> <yourhostname> -->Settings. Now scroll down to the SSH Public keys section and then remove the keys and save it.

Now you should be able to login to your server using ssh as usual.

Tech Notes:- Got fatal error 1236 from master when reading data from binary log: 'Cannot replicate anonymous transaction when AUTO_POSITION = 1

 Last_IO_Errno: 1236
                Last_IO_Error: Got fatal error 1236 from master when reading data from binary log: 'Cannot replicate anonymous transaction when AUTO_POSITION = 1, at file ../binlogs/mysql-bin.000006, position 194.; the first event '' at 4, the last event read from '../binlogs/mysql-bin.000006' at 259, the last byte read from '../binlogs/mysql-bin.000006' at 259.'
               Last_SQL_Errno: 1396
 Master_UUID: ab7c6739-be64-11ea-b478-080027fd2c56

Retrieved_Gtid_Set: ab7c6739-be64-11ea-b478-080027fd2c56:1-29

            Executed_Gtid_Set: 846dc749-bf34-11ea-852d-08002749f9b4:1-1032,

ab7c6739-be64-11ea-b478-080027fd2c56:1-4:30-114202

On master check the binary log 'mysql-bin.000006' at position 194

mysql> show binlog events in 'mysql-bin.000006' from 4 limit 5;
+------------------+-----+----------------+-----------+-------------+---------------------------------------------------------------------------------------------------------------+
| Log_name         | Pos | Event_type     | Server_id | End_log_pos | Info                                                                                                          |
+------------------+-----+----------------+-----------+-------------+---------------------------------------------------------------------------------------------------------------+
| mysql-bin.000006 |   4 | Format_desc    |        55 |         123 | Server ver: 5.7.30-33-log, Binlog ver: 4                                                                      |
| mysql-bin.000006 | 123 | Previous_gtids |        55 |         194 | ab7c6739-be64-11ea-b478-080027fd2c56:1-29                                                                                                    |

+------------------+-----+----------------+-----------+-------------+---------------------------------------------------------------------------------------------------------------+

The solution is to skip the transactions associated with GTID ab7c6739-be64-11ea-b478-080027fd2c56:1-29 
after making sure that this has been replicated already if not, we need to find and apply the transactions manually.

SET GTID_NEXT='ab7c6739-be64-11ea-b478-080027fd2c56:1';
BEGIN;COMMIT;
SET GTID_NEXT='ab7c6739-be64-11ea-b478-080027fd2c56:2';
BEGIN;COMMIT;
SET GTID_NEXT='ab7c6739-be64-11ea-b478-080027fd2c56:3';
BEGIN;COMMIT;
SET GTID_NEXT='ab7c6739-be64-11ea-b478-080027fd2c56:4';
BEGIN;COMMIT;
SET GTID_NEXT='ab7c6739-be64-11ea-b478-080027fd2c56:5';
BEGIN;COMMIT;
SET GTID_NEXT='ab7c6739-be64-11ea-b478-080027fd2c56:6';
BEGIN;COMMIT;
SET GTID_NEXT='ab7c6739-be64-11ea-b478-080027fd2c56:7';
BEGIN;COMMIT;
SET GTID_NEXT='ab7c6739-be64-11ea-b478-080027fd2c56:8';
BEGIN;COMMIT;
SET GTID_NEXT='ab7c6739-be64-11ea-b478-080027fd2c56:9';
BEGIN;COMMIT;
SET GTID_NEXT='ab7c6739-be64-11ea-b478-080027fd2c56:10';
BEGIN;COMMIT;
SET GTID_NEXT='ab7c6739-be64-11ea-b478-080027fd2c56:11';
BEGIN;COMMIT;
SET GTID_NEXT='ab7c6739-be64-11ea-b478-080027fd2c56:12';
BEGIN;COMMIT;
SET GTID_NEXT='ab7c6739-be64-11ea-b478-080027fd2c56:13';
BEGIN;COMMIT;
SET GTID_NEXT='ab7c6739-be64-11ea-b478-080027fd2c56:14';
BEGIN;COMMIT;
SET GTID_NEXT='ab7c6739-be64-11ea-b478-080027fd2c56:15';
BEGIN;COMMIT;
SET GTID_NEXT='ab7c6739-be64-11ea-b478-080027fd2c56:16';
BEGIN;COMMIT;
SET GTID_NEXT='ab7c6739-be64-11ea-b478-080027fd2c56:17';
BEGIN;COMMIT;
SET GTID_NEXT='ab7c6739-be64-11ea-b478-080027fd2c56:18';
BEGIN;COMMIT;
SET GTID_NEXT='ab7c6739-be64-11ea-b478-080027fd2c56:19';
BEGIN;COMMIT;
SET GTID_NEXT='ab7c6739-be64-11ea-b478-080027fd2c56:20';
BEGIN;COMMIT;
SET GTID_NEXT='ab7c6739-be64-11ea-b478-080027fd2c56:21';
BEGIN;COMMIT;
SET GTID_NEXT='ab7c6739-be64-11ea-b478-080027fd2c56:22';
BEGIN;COMMIT;
SET GTID_NEXT='ab7c6739-be64-11ea-b478-080027fd2c56:23';
BEGIN;COMMIT;
SET GTID_NEXT='ab7c6739-be64-11ea-b478-080027fd2c56:24';
BEGIN;COMMIT;
SET GTID_NEXT='ab7c6739-be64-11ea-b478-080027fd2c56:25';
BEGIN;COMMIT;
SET GTID_NEXT='ab7c6739-be64-11ea-b478-080027fd2c56:26';
BEGIN;COMMIT;
SET GTID_NEXT='ab7c6739-be64-11ea-b478-080027fd2c56:27';
BEGIN;COMMIT;
SET GTID_NEXT='ab7c6739-be64-11ea-b478-080027fd2c56:28';
BEGIN;COMMIT;
SET GTID_NEXT='ab7c6739-be64-11ea-b478-080027fd2c56:29';
BEGIN;COMMIT;
SET GTID_NEXT='AUTOMATIC';

A script may be programmed to generate the above statements
Now we should be able to start the slave
start slave;