Skip to content

Eureka Clustering documentation and best practices #203

Open
@aivans

Description

@aivans

I understand that there is a section in the documentation about Eureka Peer Awareness, but somehow I do not find this enough.
I find it hard to understand how this would make Eureka more HA. There would be a chain server1, server2, server3 and they are cross referenced, but if one node fails the chain breaks anyway.
Would it be a good option to just specify a list containing all 3 servers separated by comma for each all of them (server1, server2, server3)?

What is your experience?

PS: did not try DNS yet

Activity

aivans

aivans commented on Feb 11, 2015

@aivans
Author

Here are my fist findings:

  • the HA strategy seems to be one main instance with backup eureka servers
  • the clients are provided with a list of eureka servers through config (or dns)
  • the clients connect to the first server from the list, and this is where they send heartbeats and fetch information
  • the other eureka servers are just sitting idle and have no registered application (unless eureka servers reference each other, and in this case only referenced eureka instances will show app, but no business clients)
  • in case server1 becomes unavailable, all the clients migrate to the next eureka server from the list
    and keep trying to contact server1
  • in case server1 comes back online, all the clients migrate back to server1

Remarks:

  • this seems to be sufficient to have HA considering that clients cache locally

Questions and concerns:

  • in case of a high number of clients, it may be that in case server1 fails, server2 may fail too because all clients will try to migrate to it
    Seems like they took care of this when enabling DNS for serviceURLs:
    // Rearrange the fail over server list to distribute the load
    arrangeListBasedonHostname(serviceUrls);
    This leads to another concern.
  • I tried to connect a few services to server1 and a few to server2 by specifying different orders in serviceUrls list. The problem is that in this case topology partitions are created. Even if the 2 eureka servers are cross referenced and they are aware of eachother they do not exchange their registered applications, so BusinessService1 connected to Server1 will have no knowledge about BusinessService2 connected to Server2. The only time the servers have same registry information is when a server is restarted manually. At this point DiscoveryClient.getAndStoreFullRegistry() is called at startup. But after a few minutes, some instances are expired becuase no heartbeats are received.

Remarks:

  • would be nice to be able to distribute clients to different Eureka servers if the servers would exchange their registries regularly, but maybe this was not the intention
  • Maybe it is working with DNS enabled since there exists the capability to connect to different servers in the list randomly
aivans

aivans commented on Feb 13, 2015

@aivans
Author

Hi again,

I am pleased to announce that the behaviour from the previous post happens because all my Eureka nodes were configured with localhost as host name.
Changing to peer1 and peer2 as in the documentation allows for the creation of a mirrored style cluster.

radenui

radenui commented on Mar 2, 2015

@radenui

Hello,

I have a similar problem here.
I'm using AWS, and I'd like to leverage the HA provided by Netflix using DNS and the region settings (https://github.com/Netflix/eureka/wiki/Configuring-Eureka-in-AWS-Cloud#configuring-eips-using-dns).
I'm not really interested in EIP or Auto-scaling groups here, just the DNS part.

As my Eureka clients will be in different Availability zones, I will need them to contact the Eureka server in their own availability zone (as a priority, with fallback on other zones in case of a problem).
And I still will need my Eureka servers to connect each other (across multiple AZs) and share their configuration.

It seems like I cannot use the same properties defined in Netflix wiki (they are not interpreted by the client).
And the way the authentication works (in the URL !!) seems a little restrictive in this case.

Do you have any suggestion ?

Thanks a lot,

Arthur

aivans

aivans commented on Mar 2, 2015

@aivans
Author
eureka:
  client:
    registerWithEureka: true
    fetchRegistry: true
    useDnsForFetchingServiceUrls: true
    eurekaServerDNSName: eureka.local
    eurekaServerPort: 8761
    eurekaServerURLContext: eureka  

The settings above definitely do the trick for me.

aivans

aivans commented on Mar 2, 2015

@aivans
Author

I did not test multiple zones though.
Please come back and share your results for your multiple zones setup.

radenui

radenui commented on Mar 3, 2015

@radenui

Thank you for your answers!

I started using these properties:

eureka:
  client:
    registerWithEureka: true
    fetchRegistry: true
    useDnsForFetchingServiceUrls: true
    eurekaServerDNSName: eureka.local
    eurekaServerPort: 8761
    eurekaServerURLContext: eureka
    region: us-east-1
    availabilityZones:
      us-east-1: us-east-1a,us-east-1b
    preferSameZoneEureka: true

On the DNS side, I have:

  • a TXT record for the region
  • one TXT record by zone (with "user:password@server" as a value).

Into my Eureka servers (wanted them to descover their peers with DNS) and it seems like the DNS discovery does not allow to define peers at all: I had a very strange and inconsistent behavior using the same client params. Any idea why ?

But this seem to work pretty well with the eureka clients:

For the availability zones, not sure what is happening: I have this error sometimes (and have no idea why):

No match for the zone us-east-1a in the list of available zones [us-east-1b]

And

Updating the serviceUrls as they seem to have changed from [http://user:password@server-us-east-1b:8761/eureka/] to [http://user:password@server-us-east-1a:8761/eureka/]

I will try to increase the log level to see what's happening here, and will keep you in touch.
Thanks a lot,

Arthur

aivans

aivans commented on Mar 3, 2015

@aivans
Author

Never tested multiple zones.
In one single zone (defaultZone) I could configure multiple servers and they saw eachother as replicas.

How are your clients configured?

radenui

radenui commented on Mar 3, 2015

@radenui

I tried the same "client" configuration for both eureka clients and servers (see previous message).
It only works on the clients.
On the Eureka servers, they are not able to use the DNS information to define peers: I had to use "defaultZone" as well.

aivans

aivans commented on Mar 3, 2015

@aivans
Author

i would start with the default region(don't try to set it yet) and default zone and see if the dns config is ok.
It is ok when in the Eureka web UI you will see servers as available replicas when they are all started.
Afterwards, I would try a different zone for one server.

My dns config:
txt.us-east-1 IN TXT "defaultZone.eureka.local"
txt.defaultZone IN TXT "user:password@peer1" "user:password@peer2"

aivans

aivans commented on Mar 3, 2015

@aivans
Author
availabilityZones:
      us-east-1: us-east-1a,us-east-1b

My hunch is that for the client you should specify one single zone, after all the client runs in a single zone, isn't it?

aivans

aivans commented on Mar 3, 2015

@aivans
Author

Actually, i think your issue is that you specify availabilityZones. you should leave that to the dns.

radenui

radenui commented on Mar 3, 2015

@radenui

Ok, I can try that.
But as I cannot set the datacenter as "Amazon", how does the eureka client know its zone ?

aivans

aivans commented on Mar 3, 2015

@aivans
Author

datacenter=cloud?

radenui

radenui commented on Mar 3, 2015

@radenui

I like that!
what is the full name for this property? just "datacenter" ?

aivans

aivans commented on Mar 3, 2015

@aivans
Author

Arthur, maybe you need to read the docs a bit, and the source code.

25 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @cforce@ryanjbaxter@spencergibb@asarkar@aivans

        Issue actions

          Eureka Clustering documentation and best practices · Issue #203 · spring-cloud/spring-cloud-netflix