This is Jay from the CloudSQL team. Thank a lot for these great questions. Let me try to help with clarifications as below. Please let me know if you have any further questions.
The failover feature is designed to provider higher than zonal availability for Cloud SQL 2nd Gen instances. Without failover replica instance a Cloud SQL 2nd Gen instance will be out of service in the unlikely event of a zone outage. A failover replica instance is required in order to be able to failover to a different zone. When the zone failure is detected, the master instance will be recreated in the zone where failover replica resides, with data from the failover replica, and the failover replica will be 'pushed out' to another healthy zone (The actual implementation under the hood is more complicated, but this is what the external user observes). In this way, the metadata of the master instance keeps unchanged before and after the failover and there should be no action in application side to be taken for events like zone failure.
You can try out the failover behavior by calling the API directly (API document: https://cloud.google.com/sql/docs/admin-api/v1beta4/instances/failover) to trigger a manual failover.
1) In 1st gen, when there was a failure a new instance was automagically spun up and activated. In 2nd gen, is this no longer the case? Do I NEED to create a Failover Replica for the same behavior?
The failover is designed to provider higher than zonal availability for Cloud SQL 2nd Gen instances. I don't think there is similar behavior implemented in 1st Gen instances.
2) In the event of a failover, will my failover replica automatically become primary? Or do I need to trigger a failover by hand when something is down?
In the event of a failover, what you'll observe is that your primary database instance will be moved to a healthy zone (the zone where failover replica resides), and the failover replica will be moved to another healthy zone. There is no change required at all in terms of how your application connects to database, assuming that your application handles database reconnection well.
Currently we triggers failover automatically when there is zone level failures. You can also try to call the failover API directly to try out failover behavior on a specific instance (https://cloud.google.com/sql/docs/admin-api/v1beta4/instances/failover).
3) How long of a bad event is there before the automatic failover process is started?
As I explained in the previous question, currently we only triggers auto-failure in case of zone level failure. The failover is triggered as soon as the zone failure is detected.
4) My failover replica has an IP.. do I need to change my clients to use this IP, or will the old primary IP now start pointing at the failover replica? i.e. is this really a floating IP that gets moved?
No. There should be zero change required in your clients. After the failover, your client still connects tot he old primary IP, which now points to the primary instance that is moved to a healthy zone.
5) Can I use my failover replica as a read slave, or must it just sit idle until an event?
Yes. A failover replica is perfectly capable of being served as a read replica.
6) What happens to the old primary in a failover after it comes back. Does it become a failover replica for the new primary, or do I need to do something by hand?
The primary stays as primary before and after the failover process. It is just moved to a healthy zone. Therefore there is no such thing as "old primary comes back" as it always there, and there is nothing need to be done by hand.
7) How do I reset my original primary to be the real master after a failover event is complete?
Same as questions 6.
